Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
Open accessAbstract
Large language models (LLMs) are increasingly used as rerankers in information retrieval, yet their ranking behavior can be steered by small, natural-sounding prompts.To expose this vulnerability, we present Rank Anything First (RAF), a two-stage token optimization method that crafts concise textual perturbations to consistently promote a target item in LLMgenerated rankings while remaining hard to detect.Stage 1 uses Greedy Coordinate Gradient to shortlist candidate tokens at the current position by combining the gradient of the rank-target with a readability score; Stage 2 evaluates those candidates under exact ranking and readability losses using an entropy-based dynamic weighting scheme, and selects a token via temperature-controlled sampling.RAF generates ranking-promoting prompts tokenby-token, guided by dual objectives: maximizing ranking effectiveness and preserving linguistic naturalness.Experiments across multiple LLMs show that RAF significantly boosts the rank of target items using naturalistic language, with greater robustness than existing methods in both promoting target items and maintaining naturalness.These findings underscore a critical security implication: LLMbased reranking is inherently susceptible to adversarial manipulation, raising new challenges for the trustworthiness and robustness of modern retrieval systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Xing et al. introduced RAF, a two-stage gradient method balancing rank gain with readability.
“Xing et al. (2026) introduced RAF, a two-stage gradient method balancing rank gain with readability.”
From GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization (2026) · cites this paper as Xing et al., 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 32
- DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024. Deepseek llm: Scaling open- source language models with longtermism. Preprint, arXiv:2401.02954. · arXiv
- Yixuan Du, Chenxiao Yu, Haoyan Xu, Ziyi Wang, Yue Zhao, and Xiyang Hu. 2026. Multimodal generative engine optimization: Rank manipula- tion for vision-language model rankers. Preprint, arXiv:2601.12263. · arXiv
- Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407.
- Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability. Preprint, arXiv:2402.08679. · arXiv
- Xiyang Hu. 2025. Dynamics of adversarial attacks on large language model-based search engines. Preprint, arXiv:2501.00745. · arXiv
- Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. · arXiv
- Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim, Minchul Yang, and Chanyoung Park. 2024. Large language models meet collaborative fil- tering: An efficient all-round llm-based recommender system. Preprint, arXiv:2404.11343. · arXiv
- Aounon Kumar and Himabindu Lakkaraju. 2024. Ma- nipulating large language models to increase product visibility. Preprint, arXiv:2404.07981. · arXiv
- Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A. Rossi, Yan Liu, Xiyang Hu, and Yue Zhao. 2026. "someone hid it": Query-agnostic black-box attacks on llm-based retrieval. Preprint, arXiv:2602.00364. · arXiv
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, and et al. 2022. Holistic evaluation of language models. Preprint, arXiv:2211.09110. · arXiv
- Weiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer, Matt Fredrikson, and Zifan Wang. 2025. Llm whisperer: An inconspicuous attack to bias llm re- sponses. Preprint, arXiv:2406.04755. · arXiv
- Qidong Liu, Xiangyu Zhao, Yuhao Wang, Yejing Wang, Zijian Zhang, Yuqi Sun, Xiang Li, Maolin Wang, Pengyue Jia, Chong Chen, Wei Huang, and Feng Tian. 2025. Large language model enhanced recommender systems: A survey. Preprint, arXiv:2412.13432. · arXiv
- Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024a. Autodan: Generating stealthy jailbreak prompts on aligned large language models. Preprint, arXiv:2310.04451. · arXiv
- Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024b. Automatic and univer- sal prompt injection attacks against large language models. Preprint, arXiv:2403.04957. · arXiv
- Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zi- hao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2024c. Prompt injection attack against llm-integrated appli- cations. Preprint, arXiv:2306.05499. · arXiv
- Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. Preprint, arXiv:2305.02156. · arXiv
- Liang-bo Ning, Shijie Wang, Wenqi Fan, Qing Li, Xin Xu, Hao Chen, and Feiran Huang. 2024. Cheatagent: Attacking llm-empowered recommender systems via llm agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, KDD '24, page 2284-2295. ACM. · DOI · Paper page
- Qiyao Peng, Hongtao Liu, Hua Huang, Yuhan Chen, Lianghao Xia, Chenxu Zhu, Zhenwei Tang, Liang Zhang, Yaochen Zhu, Jianxin Li, and Xiangnan He. 2025. A survey on llm-powered agents for recom- mender systems. Preprint, arXiv:2502.10050. · arXiv
- Samuel Pfrommer, Yatong Bai, Tanmay Gautam, and Somayeh Sojoudi. 2024. Ranking manipula- tion for conversational search engines. Preprint, arXiv:2406.03589. · arXiv
- Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. Preprint, arXiv:2101.05667. · arXiv
- Rui Pu, Chaozhuo Li, Rui Ha, Litian Zhang, Lirong Qiu, and Xi Zhang. 2024. BaitAttack: Alleviating intention shift in jailbreak attacks via adaptive bait crafting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15654-15668, Miami, Florida, USA. Associa- tion for Computational Linguistics. · DOI
- Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effec- tive text rankers with pairwise ranking prompting. In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 1504-1518, Mexico City, Mexico. Association for Computational Lin- guistics. · DOI
- Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. Preprint, arXiv:2308.03825. · arXiv
- Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2025. Optimization-based prompt injection attack to llm- as-a-judge. Preprint, arXiv:2403.17710. · arXiv
- Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542. · arXiv
- Yiming Tang, Yi Fan, Chenxiao Yu, Tiankai Yang, Yue Zhao, and Xiyang Hu. 2025. Stealthrank: Llm rank- ing manipulation via stealthy prompt optimization. Preprint, arXiv:2504.05804. · arXiv
- Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. 2024. A survey on large language models for recom- mendation. Preprint, arXiv:2305.19860. · arXiv
- Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey. Preprint, arXiv:2407.04295. · arXiv
- Jinghao Zhang, Yuting Liu, Qiang Liu, Shu Wu, Guib- ing Guo, and Liang Wang. 2024. Stealthy attack on large language model based recommendation. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5839-5857, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Bar- row, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Interpretable gradient- based adversarial attacks on large language models. Preprint, arXiv:2310.15140. · arXiv
- Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Be- yond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 2: Short Papers), pages 358-370, Mexico City, Mexico. Association for Computational Linguistics. · DOI
- Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Univer- sal and transferable adversarial attacks on aligned language models. Preprint, arXiv:2307.15043. · arXiv
Cited by
Sorted by citations
GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization
Nimase, O., Chen, Z., Zhao, Y., & Hu, X. (2026) · arXiv · doi:10.48550/arxiv.2605.29107
Self-citation
- Authors
- Xing, T., Li, J., Du, Y., & Hu, X.
- Affiliations
- National University of Singapore (SG)
- University of Southern California (US)
- Georgetown University (US)
- Arizona State University (US)
- ORCID
- Li, Jerry
- Venue
- Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2026
- Pages
- 9120-9132
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4415317037
- Cited by
- 3
- References
- 0
- Citations by year
- 2026: 3
- FWCI
- 12.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2137603, 2138259, 2138296, 2138307, 2138286)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, re-ranking, information retrieval, two-stage optimization, prompt engineering, readability, adversarial robustness, LLM security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
