Ranking Manipulation for Conversational Search Engines
Open accessAbstract
Major search engine providers are rapidly incorporating Large Language Model (LLM)-generated content in response to user queries. These conversational search engines operate by loading retrieved website text into the LLM context for summarization and interpretation. Recent research demonstrates that LLMs are highly vulnerable to jailbreaking and prompt injection attacks, which disrupt the safety and quality goals of LLMs using adversarial strings. This work investigates the impact of prompt injections on the ranking order of sources referenced by conversational search engines. To this end, we introduce a focused dataset of real-world consumer product websites and formalize conversational search ranking as an adversarial problem. Experimentally, we analyze conversational search rankings in the absence of adversarial injections and show that different LLMs vary significantly in prioritizing product name, document content, and context position. We then present a tree-of-attacks-based jailbreaking technique which reliably promotes low-ranked products. Importantly, these attacks transfer effectively to state-of-the-art conversational search engines such as perplexity$.$ai. Given the strong financial incentive for website owners to boost their search ranking, we argue that our problem formulation is of critical importance for future robustness work.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper formalizes adversarial ranking manipulation for conversational search engines and introduces the RAGDOLL dataset of real consumer product websites. It shows that prompt injections can reliably promote low-ranked products, and that these attacks transfer to perplexity.ai’s Sonar Large Online model.
Key findings
- Conversational search rankings are shaped by product name, document content, and input context position, but their relative influence varies by LLM: GPT-4 Turbo and Llama 3 are heavily influenced by product names, GPT-4 Turbo is minimally influenced by retrieved documents, and all tested LLMs tend to prefer product-document pairs earlier in the context.
- A tree-of-attacks-based prompt injection reliably boosts low-ranked products. Table 1 reports mean ranking-score gains of 3.38 (57.53%) for GPT-3.5 Turbo, 5.00 (82.94%) for GPT-4 Turbo, 6.02 (95.74%) for Llama 3 70B, 4.13 (76.23%) for Mixtral 8x22, and 2.89 (54.23%) for Sonar Large Online.
- Attacks transfer to perplexity.ai’s Sonar Large Online model: promoted products’ rankings increased by an average of almost 3 positions and more than half the gap to the top rank.
- The RAGDOLL dataset contains 1,147 webpages across five product groups and ten categories per group; the controlled experimental subset uses exactly 8 brands per product and one product model per brand.
Methods
- The paper formalizes conversational search ranking as an adversarial prompt-injection problem: documents are retrieved for a query, formatted into an LLM prompt, and products are scored by the order in which they are mentioned, with the first-mentioned product receiving the highest score and unmentioned products receiving 0.
- RAGDOLL was collected from manufacturer-hosted consumer product pages using LLM-generated brand/model lists, Google Custom Search, rule-based and LLM-based filtering, and human inspection; experiments use a controlled subset of 8 brands per product category with one model per brand.
- Ranking tendencies are estimated over randomized prompt templates and LLM responses, using F-statistics to compare the influence of product identity, source document, and input context position. Adversarial injections are generated with a TAP-style tree-of-attacks procedure using branching, evaluation over m = 2 recommender responses, pruning, termination tolerance δ = 1, max width and depth of 5, and 3 root nodes; transfer experiments host manipulated pages and query perplexity.ai’s Sonar Large Online model via URLs.
Limitations
- The paper states that the attack is not completely effective, though the vast majority of promoted products experience significantly improved rankings; cost constraints required an inexpensive evaluation step with only m = 2 recommendation LLM responses and a shallow tree depth.
- The paper states that it does not evaluate proposed defensive approaches, and that the perplexity.ai full search tool evidence is anecdotal and ephemeral because API access to the full search tool is unavailable and its implementation may change.
- Evident from the text: the main experiments use a controlled subset of 8 brands per product and one model per brand, and the attack assumes the adversarial document is retrieved for the same queries as the unmodified document; Appendix A provides only preliminary evidence that injections minimally alter embeddings (median original-adversarial similarity 0.93 vs. 0.47 for arbitrary unperturbed pairs).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
9 of 14 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured
- Premisecitation
Conversational search engines are fundamentally based on the Retrieval-Augmented Generation (RAG) architecture.
“Conversational search engines are fundamentally based on the Retrieval-Augmented Generation (RAG) architecture.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Lewis et al., 2020 source
The Introduction (p. 2) defines RAG, and the Results (pp. 5–6) evaluate it on question-answering tasks. Related Work (p. 9) mentions retrieved text in knowledge-intensive dialogue, but our copy does not say that conversational search engines are fundamentally based on RAG. I searched for conversational search and dialogue as well as the claim’s wording. (LLM, conf=0.76)
Experimental — accuracy not yet measured
- Premisecitation
The ranking of mentioned products is often critical to consumer purchasing decisions.
“We specifically consider the domain of consumer products, in which the ranking of mentioned products is often critical to consumer purchasing decisions (Yao et al., 2021).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Yao et al., 2021 source
The Introduction (p. 1) links e-commerce search to customer engagement and conversion. Section 3.1 (p. 4) says that higher-ranked search results receive more clicks. These passages support the importance of product order in online shopping, but they do not establish that the ranking of mentioned products is often critical to purchasing decisions. (LLM, conf=0.82)
Experimental — accuracy not yet measured
- Findingown data
GPT-4 Turbo is more vulnerable than GPT-3.5 to the adversarial injection procedure.
“Notably, the increased vulnerability of GPT-4 Turbo over GPT-3.5 demonstrates that improved model capabilities do not result in inherent robustness.”
Quote in the paperSubstantial claim - Findingown data
The natural ranking distributions computed in Section 5.1 can be adversarially manipulated via a prompt injection attack.
“This section provides evidence that the natural ranking distributions computed in Section 5.1 can be adversarially manipulated via a prompt injection attack.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Findingown data
The approach transfers adversarial injections targeting GPT-4 Turbo to the corresponding hosted website.
“We demonstrate the flexibility of our approach by transferring adversarial injections targeting GPT-4 Turbo in Section 5.2 to the corresponding hosted website.”
Quote in the paperSubstantial claim
- Findingown data
The CHUWI document ranks poorly for almost all featured products.
“For instance, Figure 2b shows that the CHUWI document ranks poorly for almost all featured products.”
Quote in the paperNot a substantial claim - Findingown data
The adversarial injection procedure is fairly effective across all models.
“Consistent with Figure 3, the adversarial injection procedure is fairly effective across all models, with Llama 3 70B being particularly vulnerable.”
Quote in the paperSubstantial claim - Findingown data
Despite the closed-source nature of perplexity.ai's RAG system, adversarial promotion is generally effective in substantially increasing the ranking score of the products of interest.
“Despite the closed-source nature of perplexity.ai'sRAG system, the adversarial promotion is still generally effective in substantially increasing the ranking score of the products of interest.”
Quote in the paperSubstantial claim - Findingnone given
GPT-4 Turbo and Llama 3 are heavily influenced by their latent knowledge of product names.
“Notably, GPT-4 Turbo and Llama 3 are heavily influenced by their latent knowledge of product names.”
Quote in the paperSubstantial claim - Findingown data
Llama 3 70B exhibits more adversarial susceptibility in accordance with its greater attention to document content.
“Nevertheless, Figure 3 does show that Llama 3 70B exhibits more adversarial susceptibility in accordance with its greater attention to document content.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Findingown data
After introducing an adversarial injection, the CHUWI tablet shifts from generally being ranked in the bottom half of search results to consistently ranking as the first result.
“After introducing an adversarial injection, the product shifts from generally being ranked in the bottom half of search results to consistently ranking as the first result.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Findingown data
Llama 3 70B is particularly vulnerable to the adversarial injection procedure.
“Consistent with Figure 3, the adversarial injection procedure is fairly effective across all models, with Llama 3 70B being particularly vulnerable.”
Quote in the paperNot a substantial claim - Findingown data
The relative importance of product names and documents is heavily dependent on the specific product category.
“Notably, the relative importance of each factor is heavily dependent on the specific product category.”
Quote in the paperSubstantial claim - Findingown data
Different LLMs vary significantly in prioritizing product name, document content, and context position when ranking conversational search results.
“Experimentally, we analyze conversational search rankings in the absence of adversarial injections and show that different LLMs vary significantly in prioritizing product name, document content, and context position.”
- Also stated in: § 1 Introduction · plain
“We disentangle the impacts of product name, document content, and context position on RAG ranking tendencies, and show that these influences vary significantly between LLMs.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingown data
Adversarially manipulated products generally climb in ranking.
“While some products prove more challenging than others to promote, the positive influence is clear, with adversarially manipulated products generally climbing in ranking (lying above the dashed diagonal line).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Findingown data
The CHUWI tablet initially had the lowest average ranking score.
“The CHUWI tablet initially had the lowest average ranking score.”
Quote in the paperNot a substantial claim - Findingown data
The adversarial attacks transfer effectively to state-of-the-art conversational search engines such as perplexity.ai.
“Importantly, these attacks transfer effectively to state-of-the-art conversational search engines such as perplexity.ai.”
- Also stated in: § 1 Introduction · plain
“Futhermore, these attacks transfer from handcrafted templating schemes to production conversational engines such as perplexity.ai.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Also stated in: § 1 Introduction · plain
- Findingown data
The paper demonstrates an effective end-to-end ranking manipulation attack on perplexity.ai.
“This section demonstrates an effective end-toend ranking manipulation attack on the popular conversational search engine perplexity.ai.”
Quote in the paperSubstantial claim - Findingown data
A tree-of-attacks-based jailbreaking technique reliably promotes low-ranked products in conversational search rankings.
“We then present a tree-of-attacks-based jailbreaking technique which reliably promotes low-ranked products.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Findingown data
GPT-4 Turbo is minimally influenced by retrieved documents.
“GPT-4 Turbo is also minimally influenced by retrieved documents.”
Quote in the paperSubstantial claim - Findingown data
Despite a recommender LLM system prompt emphasizing that best products should be referenced first, all LLMs are significantly influenced by input context position, tending to prefer product-document pairs earlier in the context.
“Despite using a recommender LLM system prompt which emphasizes that best products should be referenced first, all LLMs are significantly influenced by the input context position, tending to prefer productdocument pairs earlier in the context (Figure 8).”
Quote in the paperSubstantial claim - Findingown data
After prepending the adversarial injection text, the promoted products consistently rank near the top of the search results.
“After prepending the adversarial injection text (second text box), the promoted products consistently rank near the top of the search results.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
- Findingown data
All models transfer a high input context position to a high ranking.
“All models transfer a high input context position to a high ranking.”
Quote in the paperSubstantial claim - Findingown data
Different LLMs vary significantly in which features most heavily influence rankings.
“We show that while all three have significant sway over product rankings, different LLMs vary significantly in which features most heavily influence rankings.”
Quote in the paperSubstantial claim - Findingown data
For the blender product category, the average Sonar Large Online ranking score for the Smeg Pastel Blue goes from 1.6 to 5.5 upon manipulation.
“average Sonar Large Online ranking score for the Smeg Pastel Blue goes from 1.6 → 5.5 upon manipulation”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
- Findingown data
Adversarial injections minimally affect document embeddings.
“Figure 4 suggests that adversarial injections minimally affect document embeddings.”
Quote in the paperSubstantial claim - Findingown data
These adversarial injections transfer from handcrafted templates to production RAG systems, as demonstrated by successfully manipulating the search results for perplexity.ai's Sonar Large Online model on self-hosted websites.
“These adversarial injections transfer from handcrafted templates to production RAG systems, as we demonstrate by successfully manipulating the search results for perplexity.ai'sSonar Large Online model on self-hosted websites.”
Quote in the paperSubstantial claim - Findingown data
The 99th percentile cosine similarity of arbitrary unperturbed document pairs is 0.71.
“the 99th percentile similarity of the arbitrary unperturbed document pairs is just 0.71”
Quote in the paperSubstantial claim - Findingown data
The vast majority of promoted products experience significantly improved rankings.
“The principle shortcoming of this work is that our attack is not completely effective, although the vast majority of promoted products experience significantly improved rankings (Figure 3).”
Quote in the paperSubstantial claim - Findingown data
Adversarial injections for Sonar Large Online are also effective against perplexity.ai's production search tool, based on anecdotal evidence.
“This section provides anecdotal evidence that our adversarial injections for Sonar Large Online are also effective against perplexity.ai'sproduction search tool.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
- Findingown data
Without an adversarial injection, the perplexity.ai search tool is hesitant to recommend a blender from within the provided options, electing instead to draw upon other sources to make a recommendation.
“Without an adversarial injection (top row), the tool is hesitant to recommend a blender from within the provided options, electing instead to draw upon other sources to make a recommendation.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
- Findingown data
The LLM can generally complete the desired 60-model list within two queries.
“We observe that the LLM can generally complete the desired 60-model list within two queries.”
Quote in the paperNot a substantial claim - Findingown data
Promoted products' rankings were increased by an average of almost 3 positions.
“Table 1 shows quantitatively that promoted products' rankings were increased by an average of almost 3 positions and more than half the gap to the top rank.”
Quote in the paperSubstantial claim - Findingown data
Mixtral 8x22 is heavily influenced by input context position.
“Note that the Mixtral 8x22 model generally has the smallest standard deviation; this matches our expectations from Figure 2d, which shows that Mixtral 8x22 is heavily influenced by input context position.”
Quote in the paperNot a substantial claim - Findingown data
The median cosine similarity of arbitrary unperturbed document pairs is 0.47.
“the median similarity of the arbitrary unperturbed document pairs is 0.47”
Quote in the paperNot a substantial claim - Findingown data
Perplexity.ai's closed-source RAG backend is vulnerable to injections.
“we are thus able to demonstrate that perplexity.ai'sclosed-source RAG backend is vulnerable to injections”
Quote in the paperSubstantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
- Findingown data
The attack is not completely effective.
“The principle shortcoming of this work is that our attack is not completely effective, although the vast majority of promoted products experience significantly improved rankings (Figure 3).”
Quote in the paperSubstantial claim - Findingown data
The Mixtral 8x22 model generally has the smallest standard deviation.
“Note that the Mixtral 8x22 model generally has the smallest standard deviation; this matches our expectations from Figure 2d, which shows that Mixtral 8x22 is heavily influenced by input context position.”
Quote in the paperNot a substantial claim - Findingown data
Promoted products' rankings were increased by more than half the gap to the top rank.
“Table 1 shows quantitatively that promoted products' rankings were increased by an average of almost 3 positions and more than half the gap to the top rank.”
Quote in the paperSubstantial claim - Findingreasoning
LLMs are generally unable to provide valid functioning URLs, despite their capability of gathering a list of brands and models.
“Since e-commerce website structures change frequently and LLMs are trained with data at least several months old, the LLMs are generally unable to provide valid functioning URLs, despite their capability of gathering a list of brands and models.”
Quote in the paperSubstantial claim- Rests on an assumption: E-commerce website structures change frequently and LLMs are trained with data at least several months old.
- Findingown data
The median cosine similarity of original-adversarial document pairs is 0.93.
“the median similarity of the original-adversarial document pairs is 0.93”
Quote in the paperSubstantial claim - Findingown data
Almost all original-adversarial document pairs have a cosine similarity of 0.8 or higher.
“Almost all the original-adversarial document pairs have a similarity of 0.8 or higher”
Quote in the paperSubstantial claim - Findingown data
The addition of the adversarial injection induces the search engine to consistently recommend the Smeg product.
“the addition of the adversarial injection induces the search engine to consistently recommend the Smeg product (bottom row).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
- Findingown data
For GPT-3.5 Turbo, adversarial manipulation yields a mean ranking score gain of 3.38 for the promoted product, corresponding to 57.53% of the gap to the maximum achievable score.
“| GPT-3.5 Turbo | 3.38 | 57.53 |”
Quote in the paperSubstantial claim - Interpretationnone given
The resulting score distribution reflects the product-document pairs preferred by the recommender LLM.
“The resulting score distribution reflects the product-document pairs preferred by the recommender LLM.”
Quote in the paperNot a substantial claim - Study design and conductown data
The adversarial injections do not significantly alter the text embeddings of the original unperturbed documents.
“Nevertheless, we provide preliminary evidence in Appendix A that our adversarial injections do not significantly alter the text embeddings of the original unperturbed documents.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
- Premisereasoning
A comprehensive treatment of adversarial techniques for conversational search engines is lacking from the literature.
“However, a comprehensive treatment of adversarial techniques for conversational search engines is distinctly lacking from the literature.”
Quote in the paperNot a substantial claim - Premisereasoning
Modern conversational engines load retrieved website text into the LLM context before answering a user query.
“Modern conversational engines are fundamentally RAG models which load retrieved website text into the LLM context before answering a user query.”
Quote in the paperSubstantial claim - Premisereasoning
Existing filters against harmful content often do not directly transfer to the search ranking scenario.
“Existing filters against harmful content (e.g.LlamaGuard) therefore often do not directly transfer to our scenario.”
Quote in the paperSubstantial claim - Interpretationown data
For categories towards the bottom-right, the LLM relies on its prior product knowledge and largely ignores the retrieved documents.
“Categories towards the bottom-right are those for which the LLM relies on its prior product knowledge and largely ignores the retrieved documents.”
Quote in the paperSubstantial claim - Interpretationown data
Improved model capabilities do not result in inherent robustness.
“Notably, the increased vulnerability of GPT-4 Turbo over GPT-3.5 demonstrates that improved model capabilities do not result in inherent robustness.”
Quote in the paperSubstantial claim - Interpretationown data
For categories towards the top-left, the LLM ignores the product names and attends to the documents.
“Conversely, categories towards the top-left are those for which the LLM ignores the product names and attends to the documents.”
Quote in the paperSubstantial claim - Interpretationreasoning
GPT-4 Turbo is strongly biased towards certain products irrespective of what information is present on their websites.
“This suggests that it is strongly biased towards certain products irrespective of what information is present on their websites.”
Quote in the paperSubstantial claim - Interpretationown data
Results for other LLMs are qualitatively similar to those for GPT-4 Turbo in the reproduced adversarial injection examples.
“results for other LLMs are qualitatively similar”
Quote in the paperNot a substantial claim - Premisereasoning
LLMs are generally incapable of providing valid accessible URLs.
“LLMs are generally incapable of providing valid accessible URLs”
Quote in the paperSubstantial claim- Rests on an assumption: E-commerce websites or webpages change or update frequently.
- Rests on an assumption: LLMs are trained with data at least several months old.
- May combine two claims. 22
- Classified as probably not a substantial claim. 39
May combine two claims.
PremisecitationEvaluating a jailbreaking attack is subjective to the point of often requiring human or LLM judges, whereas product ranking order is precise and quantitative.
“Evaluating a jailbreaking attack is subjective to the point of often requiring human (Zhu et al., 2023) or LLM (Mehrotra et al., 2023) judges, whereas product ranking order is precise and quantitative.”
Quote in the paperSubstantial claim2 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPrevious work has shown anecdotal evidence of prompt injection leading to product promotion for RAG models.
“Previous work has shown anecdotal evidence of prompt injection leading to product promotion for RAG models (Greshake et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Greshake et al., 2023 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationTree of Attacks with Pruning (TAP) involves iteratively expanding a tree wherein each node contains an adversarial injection attempt and associated metadata.
“upon the recent Tree of Attacks with Pruning (TAP) jailbreak (Mehrotra et al., 2023).TAP involves iteratively expanding a tree wherein each node contains an adversarial injection attempt and some associated metadata.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Mehrotra et al., 2023 source
Classified as probably not a substantial claim.
PremisecitationRAG models augment LLMs with an information retrieval mechanism that concatenates input prompts with relevant text retrieved from a vector index.
“RAG models augment LLMs with an information retrieval mechanism that concatenates input prompts with relevant text retrieved from a vector index (Lewis et al., 2020).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Lewis et al., 2020 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe paper does not evaluate partially-effective defensive approaches proposed in the literature.
“Thus while a few partially-effective defensive approaches have been proposed in the literature, we do not evaluate them here (Yi et al., 2023;Piet et al., 2023;Chen et al., 2024;Wallace et al., 2024).”
Quote in the paperNot a substantial claim2 of 3 sources checkedMay combine two claims.
Findingown dataAmong the considered LLMs, Llama 3 70B features a surprisingly bimodal distribution, while GPT-4 Turbo particularly attends to the product name.
“Among the considered LLMs, Llama 3 70B features a surprisingly bimodal distribution, while GPT-4 Turbo particularly attends to the product name.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe trend of adversarial manipulation holds across all LLMs: even though GPT and Mixtral models are minimally influenced by unmanipulated documents, they are still susceptible to adversarial injections.
“Interestingly, this trend holds across all LLMs: even though the GPT and Mixtral models are minimally influenced by unmanipulated documents (Figure 2d), they are still susceptible to adversarial injections.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The attacker's document is assumed to be selected from the vector index.
- The authors state this as an assumption: The attacker minimally edits the target document by prepending an adversarial prompt.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataThe full perplexity.ai tool exhibits similar vulnerabilities to the Sonar Large Online model, as shown anecdotally in Appendix D, though not quantified rigorously.
“Appendix D shows anecdotally that the full perplexity.aitool exhibits similar vulnerabilities to the Sonar Large Online model, although we are unable to quantify this rigorously.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone givenThe LLM is less likely to hallucinate when required to provide reasons for its answer.
“Observing that the LLM is less likely to hallucinate when required to provide reasons for its answer, we use the following prompt:”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor GPT-4 Turbo, adversarial manipulation yields a mean ranking score gain of 5.00 for the promoted product, corresponding to 82.94% of the gap to the maximum achievable score.
“| GPT-4 Turbo | 5.00 | 82.94 |”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe data collection pipeline starts with 20 brands and 3 model entries per brand for every product.
“our data collection pipeline starts with 20 brands and 3 model entries per brand for every product”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Tefal website has an average score of 1.7.
“Tefal (average score 1.7)”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenIdentifying catalog pages and third-party websites is more challenging.
“However, identifying catalog pages and third-party websites is more challenging.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingown dataThe search tool is biased towards websites with a high context position.
“the search tool in our experience is biased towards websites with a high context position”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAll three factors-product name, supporting document, and input context position-have significant sway over product rankings.
“We show that while all three have significant sway over product rankings, different LLMs vary significantly in which features most heavily influence rankings.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe Huawei and Samsung tablets tend to rank highly, whereas the CHUWI tablet ranks the lowest.
“The Huawei and Samsung tablets tend to rank highly, whereas the CHUWI tablet ranks the lowest.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingreasoningThe remaining sources are retrieved from other search results.
“The remaining sources are retrieved from other search results.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenMany websites of the unofficial and non-product types can be straightforwardly filtered with rule-based criteria, which is faster and cheaper than relying on an LLM.
“Many websites of the former two types can be straightforwardly filtered with rule-based criteria, which is faster and cheaper than relying on an LLM.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor Mixtral 8x22, adversarial manipulation yields a mean ranking score gain of 4.13 for the promoted product, corresponding to 76.23% of the gap to the maximum achievable score.
“| Mixtral 8x22 | 4.13 | 76.23 |”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataEach product includes at least 9 brands and 1-3 model entries per brand.
“Each product includes at least 9 brands and 1-3 model entries per brand.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenSometimes the LLM reports less than 20 brands.
“Sometimes the LLM reports less than 20 brands.In this case, we query it again with the same prompt but additionally instruct it to exclude the brands from the first query.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor Llama 3 70B, adversarial manipulation yields a mean ranking score gain of 6.02 for the promoted product, corresponding to 95.74% of the gap to the maximum achievable score.
“| Llama 3 70B | 6.02 | 95.74 |”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor Sonar Large Online, adversarial manipulation yields a mean ranking score gain of 2.89 for the promoted product, corresponding to 54.23% of the gap to the maximum achievable score.
“| Sonar Large Online | 2.89 | 54.23 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe RAGDOLL dataset includes 5 product groups and 10 categories per group.
“The RAGDOLL dataset includes 5 product groups and 10 categories per group.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe CHUWI document ranks poorly for most featured products, whereas the Samsung product is highly ranked when paired with any document beside the CHUWI document.
“The CHUWI document ranks poorly for most featured products, whereas the Samsung product is highly ranked when paired with any document beside the CHUWI document.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
FindingreasoningExtracting all textual content with BeautifulSoup yields a substantial amount of extraneous text unrelated to the product of interest.
“This yields a substantial amount of extraneous text which is unrelated to the product of interest (e.g., header and footer, navigation links).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe number of remaining brands and model entries after the filtering pipeline varies across products.
“the number of remaining brands and model entries after the filtering pipeline varies across products”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone givenAs of May 2024, the perplexity.ai web interface does not seem to incorporate more than 3 provided URLs as sources.
“As of May 2024, the perplexity.aiweb interface does not seem to incorporate more than 3 provided URLs as sources.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone givenRule-based filtering is efficient and effective but struggles to identify more complex undesired cases, such as catalog pages that list or compare numerous products.
“While rule-based filtering is efficient and effective, it struggles to identify more complex undesired cases, such as catalog pages that list or compare numerous products.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone givenThe paper precisely formulates the adversarial prompt injection objective and presents a jailbreaking technique to reliably boost the ranking of an arbitrary product.
“For the second question, we precisely formulate the adversarial prompt injection objective and present a jailbreaking technique to reliably boost the ranking of an arbitrary product.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe product-document pair shown had the lowest average ranking before the prompt injection.
“This product-document pair had the lowest average ranking before the prompt injection (Figure 7).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Breville website has an average score of 5.7.
“Breville (average score 5.7)”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone givenRule-based filtering can exclude common third-party sites but may not identify smaller or more specialized platforms.
“Additionally, while rule-based filtering can exclude common third-party sites, it may not identify smaller or more specialized platforms.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenIn rare cases, the brand name does not appear in the URL even when the website is official.
“In rare cases, the brand name does not appear in the URL even when the website is official.These corner cases are handled by an LLM.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe first two responses with the adversarial injection verbatim reproduce the prompt's request.
“the first two responses with the adversarial injection even verbatim reproduce the prompt's request”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationTraditional search engines algorithmically rank search output, generally employing some variation of the tf-idf weighting scheme.
“Traditional search engines algorithmically rank search output, generally employing some variation of the tf-idf weighting scheme (Ramos et al., 2003).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ramos et al., 2003
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe traditional Search Engine Optimization industry alone is valued at upwards of $80 billion.
“the traditional Search Engine Optimization (SEO) industry alone is valued at upwards of $80 billion (Lewandowski and Schultheiß, 2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lewandowski and Schultheiß, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based).
Study design and conductreasoningThe URL itself does not bias engine ranking decisions because random strings are used as webpage names.
“We ensure that the URL itself does not bias engine ranking decisions by using random strings as webpage names: e.g.,”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationJailbreaking scenarios often involve isolated users attempting to induce harmful content, whereas the search ranking scenario carries significant financial implications for large organizations.
“Jailbreaking scenarios often involve isolated users attempting to induce harmful content, whereas our search ranking scenario carries significant financial implications for large organizations.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone givenConversational search engines are black-box and feature no principled or interpretable mechanism for ranking their outputs.
“Conversely, conversational search engines are black-box and feature no principled or interpretable mechanism for ranking their outputs.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
PremisereasoningIt is generally unclear upon human inspection of recommendation output whether a model has been deceived, as without access to the unmanipulated documents it is unknown what the correct ordering should be.
“It is generally unclear upon human inspection of recommendation output whether a model has been deceived, as without access to the unmanipulated documents it is unknown what the”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe attack on Sonar Large Online does not fully comprise an attack on perplexity.ai's search tool because the actual search tool incorporates a proprietary system prompt and other potential innovations.
“this does not fully comprise an attack on perplexity.ai'ssearch tool; the actual search tool incorporates a proprietary system prompt and other potential innovations.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationCompared with other LLMs such as GPT-3.5 Turbo, the training data of GPT-4 Turbo is more recent as of this work, making it more likely to provide up-to-date product models.
“Compared with other LLMs such as GPT-3.5 Turbo, the training data of GPT-4-Turbo is more recent as of this work, making it more likely to provide up-to-date product models.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites OpenAI, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe authors publicly release their data collection and experimental source code.
“We publicly release our data collection (Section 4) and experimental (Section 5) source code.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe evidence for effectiveness against perplexity.ai's production search tool is both anecdotal and ephemeral.
“We emphasize that this is both anecdotal and ephemeral.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningCost constraints required a relatively inexpensive evaluation step in the tree-of-attacks implementation (only m = 2 recommendation LLM responses) and a shallow tree depth.
“As we computed our attacks across 50 promoted products for each LLM, cost constraints required a relatively inexpensive evaluation step in our tree-of-attacks implementation (only m = 2 recommendation LLM responses) and a shallow tree depth.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationThe authors download all webpages for the final experimental dataset from the Common Crawl.
“we download all webpages for our final experimental dataset from the Common Crawl (Crawl, 2024).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Crawl, 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: E-commerce websites or webpages change or update frequently.
May combine two claims.
Interpretationnone givenThe work focuses explicitly on search result reordering in the consumer product setting, where the primary effects of an attack are to provide users with inferior recommendations.
“We note that our work focuses explicitly on search result reordering in the consumer product setting, where the primary effects of an attack are to provide users with inferior recommendations.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe authors release the full RAGDOLL dataset in the format of product page URLs to the public under the CC-BY-4.0 license.
“We release the full dataset in the format of product page URLs, to the public under the CC-BY-4.0 license.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Study design and conductnone givenThe fetching, searching, and filtering process inherently guarantees that all URLs are valid and accessible.
“Our fetching, searching, and filtering process inherently guarantees that all URLs are valid and accessible.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe final dataset is post-processed via human inspection to ensure that catalog page and third-party website cases are thoroughly checked.
“Our final dataset is post-processed via a human inspection to ensure that these cases are all thoroughly checked.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe implementation of perplexity.ai's search tool is subject to change, limiting long-term reproducibility of these results.
“the implementation of its search tool is of course subject to change, limiting long-term reproducibility of these results.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe generated adversarial injection explicitly instructs the LLM how to respond, which provides clues that the perplexity.ai search tool was adversarially manipulated.
“the generated adversarial injection also explicitly instructs the LLM how to respond, which will provide clues that the perplexity.aisearch tool was indeed adversarially manipulated.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe authors additionally release the HTML source code and the extracted text for the experimental subset under Common Crawl's terms of use.
“We additionally release the HTML source code and the extracted text for this subset under Common Crawl's terms of use.”
Quote in the paperNot a substantial claim
The authors state this as an assumption.
The attacker's document is assumed to be selected from the vector index.
“Note that our problem setting focuses on the prompt-injection setting where the attacker's document is assumed to be selected from the vector index.”
The authors state this as an assumption.
The attacker minimally edits the target document by prepending an adversarial prompt.
“We specifically consider a setting in which d * is minimally edited by prepending an adversarial prompt a such that the expected ranking of p * is maximized:”
The paper relies on this without support.
E-commerce website structures change frequently and LLMs are trained with data at least several months old.
“Since e-commerce website structures change frequently and LLMs are trained with data at least several months old, the LLMs are generally unable to provide valid functioning URLs, despite their capability of gathering a list of brands and models.”
The authors state this as an assumption.
The problem setting assumes that if a user query retrieves an unmodified document d*, it generally also retrieves the adversarially perturbed document a⊕d*.
“Our problem setting assumes that if a user query retrieves an unmodified document d * , then it generally also retrieves the adversarially perturbed document a ⊕ d * .”
The paper relies on this without support.
For most retrieval systems, the condition that a query retrieves both an unmodified document and its adversarially perturbed version amounts to the two documents having text embeddings with high cosine similarity.
“For most retrieval systems, this amounts to a ⊕ d * and d * having text embeddings with a high cosine similarity.”
The paper relies on this without support.
API access to the full perplexity.ai search tool is not offered.
“API access to the full perplexity.aisearch tool is not offered.”
The paper relies on this without support.
E-commerce websites or webpages change or update frequently.
“e-commerce webpages update regularly”
“e-commerce websites change frequently”
The paper relies on this without support.
LLMs are trained with data at least several months old.
“LLMs are trained with data at least several months old”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing C-SEO methods are often tested only for a limited breadth of application domains.
“However, they are often tested only for a limited breadth of application domains; we do not know whether certain C-SEO methods would be effective for a broad range of domains.”
From C-SEO Bench: Does Conversational SEO Work? (2025) · cites this paper as Pfrommer et al., 2024
The Introduction (p. unavailable) says this study specifically considers consumer products, and §4 Dataset (p. unavailable) describes tests across 50 product categories within that domain. This supports a limited application-domain scope for the cited method, but the paper does not establish that existing C-SEO methods are often tested with similarly limited breadth. (LLM, conf=0.85)
Experimental — accuracy not yet measuredExisting evaluations consider only a single-actor scenario where only one web document adopts a C-SEO method.
“Moreover, existing evaluations consider only a single-actor scenario where only one web document adopts a C-SEO method”
From C-SEO Bench: Does Conversational SEO Work? (2025) · cites this paper as Pfrommer et al., 2024
Pfrommer et al. adapted jailbreaking techniques to ranking via black-box prompt search.
“Pfrommer et al. (2024) adapted jailbreaking techniques to ranking via black-box prompt search.”
From GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization (2026) · cites this paper as Pfrommer et al., 2024
A parallel line of work examines prompt injection and adversarial manipulation of LLM rankers, showing that model outputs can be systematically perturbed; these studies target exploitative rather than legitimate visibility.
“A parallel line of work examines prompt injection and adversarial manipulation of LLM rankers (Kumar and Lakkaraju, 2024; Pfrommer et al., 2024), showing that model outputs can be systematically perturbed; these studies, however, target exploitative rather than legitimate visibility.”
From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Pfrommer et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Attack placement is described as both prepending and appending
MinorThe attack formulation says the adversarial prompt is prepended to the document, but the attacking-LLM instructions say it should be appended. These specify opposite placements for the injection.
“d * is minimally edited by prepending an adversarial prompt a”
“You should return an adversarial suffix which , when appended to the document , consistently influences the RAG system to list the provided consumer product first”
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2× · see citing works
“The advent of powerful open-source LLMs means that malicious users can generate harmful content relatively easily on rented hardware, limiting the incentive to jail-break commercial models (Touvron et al., 2023).”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“A subsequent fine-tuning stage attempts to align the model with human intentions by limiting the generation of objectionable content and improving instruction-following performance (Ouyang et al., 2022).”
BackgroundInfluentialRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 3× · see citing works
“For retrieval, important innovations include end-to-end retrieval fine-tuning (Lewis et al., 2020), query rewriting (Ma et al., 2023), and hypothetical document generation (Gao et al., 2023a).”
BackgroundMethodMaskery, H., & Meads, J. (1992) · ACM SIGCHI Bulletin · doi:10.1145/142386.142389 · Cited by 964
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · cited 2× · see citing works
“One important concept in response generation is that of reranking , whereby retrieved information is relocated to the edges of the input context (Gao et al., 2023b).”
BackgroundMethodGreshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667 · cited 2× · see citing works
“While jailbreaking attacks manipulate inputs fed directly through a user interface, prompt injections instead exploit the blurred distinction between instructions and data in the LLM context.”
BackgroundJailbreaking Black Box Large Language Models in Twenty Queries
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00010 · Cited by 523 · cited 1× · see citing works
“Our work investigates a few fundamental factors driving conversational search rankings and provides evidence that these rankings are susceptible to adversarial manipulation (see Figure 1). refine adversarial strings (Chao et al., 2023; Perez et al., 2022; Wu et al., 2023; Mehrotra et al., 2023).”
BackgroundJailbreaking Black Box Large Language Models in Twenty Queries
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023) · arXiv · doi:10.48550/arxiv.2310.08419 · Cited by 522 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
A Theoretical Framework for Conversational Search
Radlinski, F., & Craswell, N. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020183 · Cited by 473 · cited 1× · see citing works
“We argue that a main application of LLM jailbreaking efforts will instead concern conversational search engines , which offer a natural-language alternative to traditional search engines (Radlinski and Craswell, 2017).”
BackgroundRed Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · cited 1× · see citing works
“Our work investigates a few fundamental factors driving conversational search rankings and provides evidence that these rankings are susceptible to adversarial manipulation (see Figure 1). refine adversarial strings (Chao et al., 2023; Perez et al., 2022; Wu et al., 2023; Mehrotra et al., 2023).”
BackgroundPrecise Zero-Shot Dense Retrieval without Relevance Labels
Gao, L., Ma, X., Lin, J., & Callan, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.99 · Cited by 362 · cited 1× · see citing works
“For retrieval, important innovations include end-to-end retrieval fine-tuning (Lewis et al., 2020), query rewriting (Ma et al., 2023), and hypothetical document generation (Gao et al., 2023a).”
Background(2018) · Fundamentals of Abstract Analysis · doi:10.1201/9781315275444-12 · Cited by 351
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292 · cited 1× · see citing works
“Recent years have seen the emergence of Large Language Models (LLMs) as highly capable conversational agents (Solaiman et al., 2019; OpenAI, 2023; Touvron et al., 2023).”
BackgroundJailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.02483 · Cited by 280 · cited 1× · see citing works
“The underlying mechanisms behind these jailbreaking methods are analyzed in Wei et al. (2024), which posits that this vulnerability stems from conflict between a model’s capabilities and safety goals as well as a failure to effectively generalize.”
BackgroundJailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3508 · Cited by 280 · see citing works
Improving Passage Retrieval with Zero-Shot Question GenerationPreprint
Sachan, D. S., Lewis, M., Joshi, M., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2204.07496 · Cited by 276 · cited 1× · see citing works
“Other related approaches incorporate pointwise (Liang et al., 2023; Sachan et al., 2022), listwise (Zhuang et al., 2023) and pairwise (Liu et al., 2023) ranking prompts.”
BackgroundIs ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., & Ren, Z. (2023) · arXiv · doi:10.48550/arxiv.2304.09542 · Cited by 270 · cited 1×
“Initial attempts showed that GPT-family models can effectively perform zero-shot passage ranking (Sun et al., 2023).”
BackgroundIs ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., & Ren, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.923 · Cited by 270 · see citing works
Automatically Auditing Large Language Models via Discrete OptimizationPreprint
Jones, E., Dragan, A., Raghunathan, A., & Steinhardt, J. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2303.04381 · Cited by 252 · see citing works
Siegel, A. F. (1994) · doi:10.1016/c2015-0-00463-4 · Cited by 202 · cited 1×
“F-statistics compute the ratio of between-group variability to within-group variability (Siegel, 2016); here, we group by the categorical variable of interest ( i , j , or c ).”
MethodUniversal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197 · cited 2× · see citing works
“Our ethical considerations are similar to those in established jailbreaking attacks (Zou et al., 2023).”
Background(2019) · Some Glad Morning · doi:10.2307/j.ctvq4c06f.22 · Cited by 189
Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., & Karbasi, A. (2023) · arXiv · doi:10.48550/arxiv.2312.02119 · Cited by 183 · see citing works
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models
Yi, J., Xie, Y., Zhu, B., Kıcıman, E., Sun, G., Xie, X., & Wu, F. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709179 · Cited by 175 · cited 3× · see citing works
“Evaluating a jailbreaking attack is subjective to the point of often requiring human (Zhu et al., 2023) or LLM (Mehrotra et al., 2023) judges, whereas product ranking order is precise and quantitative.”
BackgroundInjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2403.02691 · Cited by 143 · cited 1×
“Various benchmarks for assessing the vulnerability of LLM-integrated systems to prompt injection attacks have also been proposed (Zhan et al., 2024; Yi et al., 2023; Toyer et al., 2024).”
BackgroundApruzzese, G., Anderson, H. S., Dambra, S., Freeman, D. M., Pierazzi, F., & Roundy, K. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00031 · Cited by 93 · see citing works
Improving Passage Retrieval with Zero-Shot Question Generation
Sachan, D., Lewis, M., Joshi, M., Aghajanyan, A., Yih, W.-T., Pineau, J., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.249 · Cited by 90 · see citing works
Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts
Wu, Y.-W., Li, X., Liu, Y.-X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.09127 · Cited by 84 · cited 1×
“Our work investigates a few fundamental factors driving conversational search rankings and provides evidence that these rankings are susceptible to adversarial manipulation (see Figure 1). refine adversarial strings (Chao et al., 2023; Perez et al., 2022; Wu et al., 2023; Mehrotra et al., 2023).”
BackgroundHard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2302.03668 · Cited by 77 · cited 1× · see citing works
“The structure of jailbreaking strings varies widely, from human-interpretable roleplay-ing prompts (Mehrotra et al., 2023) to ASCII art (Jiang et al., 2024) and seemingly random text produced by discrete optimization over tokens (Wen et al., 2024; Zou et al., 2023).”
BackgroundHijacking Large Language Models via Adversarial In-Context LearningPreprint
Zhou, X.-Y., Qiang, Y., Zade, S. Z., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.09948 · Cited by 58 · cited 1× · see citing works
“These attacks target LLM-integrated applications by injecting adversarial text into external data that is retrieved for the LLM (Liu et al., 2023; Qiang et al., 2023).”
BackgroundLlama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.06674 · Cited by 56 · cited 1× · see citing works
“This is especially true for approaches that attempt to reflect on the model response (Inan et al., 2023).”
BackgroundPublic awareness and attitudes towards search engine optimization
Lewandowski, D., & Schultheiß, S. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2022.2056507 · Cited by 55 · see citing works
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Piet, J., Alrashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D. (2023) · arXiv · doi:10.48550/arxiv.2312.17673 · Cited by 54
Learning a Product Relevance Model from Click-Through Data in E-Commerce
Yao, S., Tan, J., Chen, X., Yang, K., Xiao, R., Deng, H., & Wan, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450129 · Cited by 33 · cited 1× · see citing works
“We specifically consider the domain of consumer products, in which the ranking of mentioned products is often critical to consumer purchasing decisions (Yao et al., 2021).”
BackgroundThe Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024) · arXiv · doi:10.48550/arxiv.2404.13208 · Cited by 14 · cited 1× · see citing works
“Thus while a few partially-effective defensive approaches have been proposed in the literature, we do not evaluate them here (Yi et al., 2023; Piet et al., 2023; Chen et al., 2024; Wallace et al., 2024).”
BackgroundFormalizing and Benchmarking Prompt Injection Attacks and Defenses
Liu, Y., Jia, Y., Geng, R., Jia, J., & Gong, N. Z. (2023) · arXiv · doi:10.48550/arxiv.2310.12815 · Cited by 14 · cited 2× · see citing works
“These attacks target LLM-integrated applications by injecting adversarial text into external data that is retrieved for the LLM (Liu et al., 2023; Qiang et al., 2023).”
BackgroundStruQ: defending against prompt injection with structured queries
Chen, S., Piet, J., Sitawarin, C., & Wagner, D. (2024) · arXiv · doi:10.48550/arxiv.2402.06363 · Cited by 10 · cited 1× · see citing works
“Thus while a few partially-effective defensive approaches have been proposed in the literature, we do not evaluate them here (Yi et al., 2023; Piet et al., 2023; Chen et al., 2024; Wallace et al., 2024).”
BackgroundArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMsPreprint
Jiang, F., Xu, Z., Niu, L., Xiang, Z. J., Ramasubramanian, B., Li, B., & Poovendran, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11753 · Cited by 8 · cited 1× · see citing works
“The structure of jailbreaking strings varies widely, from human-interpretable roleplay-ing prompts (Mehrotra et al., 2023) to ASCII art (Jiang et al., 2024) and seemingly random text produced by discrete optimization over tokens (Wen et al., 2024; Zou et al., 2023).”
BackgroundBenchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., & Wu, F. (2023) · arXiv · doi:10.48550/arxiv.2312.14197 · Cited by 8 · see citing works
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2023) · arXiv · Cited by 7 · cited 1× · see citing works
“This workflow enables access to a dynamic knowledge base not seen during training, reduces the necessary LLM context length, and mitigates model hallucinations (Vu et al., 2023).”
BackgroundTensor Trust: Interpretable Prompt Injection Attacks from an Online Game
Toyer, S., Watkins, O., Mendes, E. A., Svegliato, J., Bailey, L., Wang, T., Ong, I., Elmaaroufi, K., Abbeel, P., Darrell, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.01011 · Cited by 7 · cited 1× · see citing works
“Various benchmarks for assessing the vulnerability of LLM-integrated systems to prompt injection attacks have also been proposed (Zhan et al., 2024; Yi et al., 2023; Toyer et al., 2024).”
Background(2023) · doi:10.48550/arxiv.2311.09735 · Cited by 4 · see citing works
Edyburn, D. L. (2009) · Journal of Special Education Technology · doi:10.1177/016264340902400406 · Cited by 1 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., & Sun, T. (2023) · arXiv · doi:10.48550/arxiv.2310.15140
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., & Goldstein, T. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2219
A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models
Zhuang, S., Zhuang, H., Koopman, B., & Zuccon, G. (2023) · arXiv · doi:10.48550/arxiv.2310.09497
Flagged first, then by citations
26 of the 75 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- PMLR.
- Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the First Instructional Conference on Machine Learning. Citeseer.
Cited by
Sorted by citations
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81 · see citing works
“…studies should explore how LLMs handle information from differ-10 ent sources in more realistic scenarios, how other information selection criteria, such as relevance and freshness (Sun et al. 2023), affect the outcomes, and how to mitigate emerging manipulation techniques (Pfrommer et al. 2024).”
Automatic Document Editing for Improved Ranking
Bardas, N., Mordo, T., Kurland, O., & Tennenholtz, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730168 · Cited by 2 · see citing works
Unveiling the Resilience of LLM-Enhanced Search Engines against Black-Hat SEO Manipulation
Chen, P., Hong, G., Wu, X., et al. (2026) · The Web Conference · doi:10.1145/3774904.3792399 · Cited by 8 · see citing works
“Perplexity reports 169M monthly visits [56], and ChatGPT officially added search capabilities in 2024 [29].”
Wang, J., Zhang, F., Chen, X., He, P., Huang, E. A., & Huang, J. X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3742423 · Cited by 1 · see citing works
“Pfrommer et al. [38] explored the impact of prompt injection attacks on conversational search engines and proposed a tree-of-attacks technique to manipulate the ranking of sources referenced by these systems.”
E-GEO: A Testbed for Generative Engine Optimization in E-Commerce
Bagga, P. S., Farias, V. F., Korkotashvili, T., Peng, T., & Wu, Y. (2025) · arXiv · doi:10.48550/arxiv.2511.20867 · Cited by 1 · see citing works
“A separate line of research examines prompt injection and manipulation attacks on LLMs (Kumar and Lakkaraju, 2024; Pfrommer et al., 2024) in a similar setting.”
Pagadala, V. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6512878
How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews
Grossman, R., Liu, S., Chen, M. K., Smith, M., Borcea, C., & Chen, Y. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809667
SAGEO Arena: A Realistic Environment for Evaluating Search-Augmented Generative Engine Optimization
Kim, S., Jeong, W., Kim, S., Lee, S., & Lee, D. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818146
Hybrid RAG Architecture: Enhancing Recommendations for Home AI Devices Control Commands
Arora, D., Yadav, V., M., R. M., et al. (2026) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct68978.2026.11700656
“The formalized model allows the system to reason about relationships among devices and be linguistically robust [3], [5], [7], [13].”
Role-Augmented Intent-Driven Generative Search Engine Optimization
Chen, X., Bao, J., Wu, H., Chen, Z., Pan, L., & Liao, Y. (2025) · arXiv · doi:10.48550/arxiv.2508.11158
“Meanwhile, a line of work [3], [6], [11], [18], [22] explores prompt injection strategies to manipulate GSE responses, but these methods might compromise semantic coherence, raising concerns regarding safety and content integrity.”
C-SEO Bench: Does Conversational SEO Work?
Puerto, H., Gubri, M., Green, T., Oh, S. J., & Yun, S. (2025) · arXiv · doi:10.48550/arxiv.2506.11097
GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization
Nimase, O., Chen, Z., Zhao, Y., & Hu, X. (2026) · arXiv · doi:10.48550/arxiv.2605.29107
- Authors
- Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S.
- Affiliations
- University of California, Berkeley (US)
- ORCID
- Sojoudi, Somayeh
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 9523-9552
- arXiv
- 2406.03589
- OpenAlex
- W4404783198
- Semantic Scholar
- f02fc3e2ab00d74f2397f70bab09a526e1910b56
- Cited by
- 8
- References
- 75
- Citations by year
- 2026: 5 · 2025: 3
- FWCI
- 2.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- prompt injection, jailbreaking, large language models, adversarial attacks, Perplexity AI, LLM-generated content
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
