The DOI matches its Crossref record; 36 of the 39 references it cites were found in a registry or an index.
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Open accessAbstract
Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines.However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking.The discrepancy between the pretraining objectives of LLMs and the ranking objective poses another challenge.In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR.Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks.Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model's ability to rank unknown knowledge.Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme.Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark.The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Rank-biased overlap (RBO)foot_6 is a widely used metric for evaluating the similarity between two ranked lists and is particularly relevant for tasks in information retrieval (Oh et al., 2022;Sun et al., 2023).”
From Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024) · cites this paper as Sun et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A representative empirical anchor is Sun et al. (2023), who show that LLMs can match or exceed strong supervised re-rankers on standard IR benchmarks, validating the 'generative engine as LLM re-ranker' abstraction we build on.”
From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Sun et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
Learning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Document Ranking with a Pretrained Sequence-to-Sequence Model
Nogueira, R. F., Jiang, Z., Pradeep, R., & Lin, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.63 · Cited by 686
Nogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Research works printed but not linked: 5
- TMLR.
- Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Few-shot learning with retrieval augmented language models. Journal of Machine Learning Re- search, 24(251):1-43.
- Nandan Thakur, Nils Reimers, Andreas Ruckl'e, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS 2021.
- Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language model with self generated instructions. In ACL 2023.
- Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. 2021. Pre-trained language model based ranking in baidu search. In KDD 2021. Method DL19 DL20 Covid NFCorpus Touche DBPedia SciFact Signal News Robust04 BEIR (Avg) BM25 50.58 47.96 59.47 30.75 44.22 31.80 67.89 33.05 39.52 40.70 43.42 Supervised train on MS MRACO monoBERT (340M) 70.50 67.28 70.01 36.88 31.75 41.87 71.36 31.44 44.62 49.35 47.16 monoT5 (220M) 71.48 66.99 78.34 37.38 30.82 42.42 73.40 31.67 46.83 51.72 49.07 monoT5 (3B) 71.83 68.89 80.71 38.97 32.41 44.45 76.57 32.55 48.49 56.71 51.36 Cohere Rerank-v2 73.22 67.08 81.81 36.36 32.51 42.51 74.44 29.60 47.59 50.78 49.45 Unsupervised instructional permutation generation ChatGPT 65.80 62.91 76.67 35.62 36.18 44.47 70.43 32.12 48.85 50.62 49.37 GPT-4 75.59 70.56 85.51 38.47 38.57 47.12 74.95 34.40 52.89 57.55 53.68 Specialized Models train on MARCO labels or ChatGPT predicted permutations MARCO Pointwise BCE 65.57 56.72 70.82 33.10 17.08 32.28 55.37 19.30 41.52 46.00 39.43 MARCO Listwise CE 65.99 57.97 66.31 32.61 20.15 30.79 37.57 18.09 38.11 39.93 35.45 MARCO RankNet 66.34 58.51 70.29 34.23 20.27 29.62 49.01 23.22 39.82 43.87 38.79 MARCO LambdaLoss 64.82 56.16 72.86 34.20 19.51 32.55 51.88 26.22 42.47 45.28 40.62 ChatGPT Listwise CE 65.39 58.80 76.29 35.73 38.19 40.24 64.49 31.37 47.61 48.00 47.74 ChatGPT RankNet 65.75 59.34 81.26 36.57 39.03 42.10 68.77 31.55 52.54 52.44 50.53 ChatGPT LambdaLoss 67.17 60.56 80.63 36.74 36.73 43.75 68.21 32.58 49.00 50.51 49.77 deberta-v3-xsmall (70M) 64.75 55.07 78.21 35.95 35.42 41.37 67.86 30.04 47.68 49.91 48.31 deberta-v3-small (142M) 67.85 58.84 78.88 36.55 36.16 40.99 66.66 30.29 49.17 49.73 48.55 deberta-v3-base (184M) 70.28 62.52 80.81 36.15 37.25 44.06 71.70 32.45 50.84 51.33 50.57 deberta-v3-large (435M) 70.66 67.15 84.64 38.48 39.27 47.36 74.18 32.53 51.19 56.55 53.03 deberta-v3-large 5K 70.93 64.32 84.43 38.66 40.72 46.28 73.88 31.93 52.24 55.89 53.00 deberta-v3-large 3K 70.79 63.91 84.21 38.73 39.83 45.74 74.41 31.92 52.29 57.42 53.07 deberta-v3-large 1K 69.90 64.81 83.38 38.94 36.65 44.46 71.96 30.19 50.73 53.74 51.26 deberta-v3-large 500 69.71 62.00 83.54 37.23 33.68 44.56 70.48 28.70 45.64 42.67 48.31 deberta-v3-large label 10K 66.61 57.26 74.36 33.94 18.09 34.95 35.35 21.38 39.00 44.94 37.75 deberta-v3-large label 5K 68.98 61.38 80.73 35.68 20.48 37.34 54.63 24.25 36.94 51.13 42.64 deberta-v3-large label 3K 67.41 60.42 79.82 35.49 24.54 37.39 47.31 23.29 39.87 50.65 42.29 deberta-v3-large label 1K 65.55 60.93 77.70 33.29 23.36 36.38 31.10 21.71 34.28 38.31 37.01 deberta-v3-large label 500 60.59 54.45 76.20 32.93 19.66 31.54 45.66 13.99 33.48 44.49 37.24 deberta-v3-large monoT5-3B 73.05 68.82 84.78 38.55 34.43 43.61 75.45 30.75 49.85 56.80 51.78 deberta-v3-large chatgpt+label 72.42 67.30 85.96 38.75 35.06 45.43 71.81 28.52 45.91 55.57 50.88 deberta-v3-base label 10k 65.66 59.84 71.63 34.65 16.53 32.59 34.65 22.64 37.60 44.02 36.79 deberta-v3-small label 10k 63.63 52.83 68.17 30.48 18.12 31.72 33.62 18.02 34.57 36.09 33.85 deberta-v3-xsmall label 10k 60.89 51.15 63.58 28.67 14.87 27.12 20.60 18.97 32.61 32.67 29.89 llama-7b 71.33 66.06 78.23 37.60 34.87 45.46 76.13 34.17 51.79 55.22 51.68 vicuna-7b 71.80 66.89 78.32 36.87 31.81 45.40 74.23 34.28 51.13 52.91 50.62 llama-7b 10k label 65.22 56.85 75.36 36.24 20.88 37.34 69.04 25.22 41.21 49.21 44.31 llama-7b 5k label 69.24 58.97 80.49 37.55 28.23 39.66 71.79 26.04 44.09 53.83 47.71
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
A survey on large language models for recommendation
Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01291-2 · Cited by 515
How Can Recommender Systems Benefit from Large Language Models: A Survey
Lin, J., Dai, X., Xi, Y., Liu, W., Chen, B., Zhang, H., Liu, Y., Wu, C., Li, X., Zhu, C., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3678004 · Cited by 242
Large Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225
Self-citationDense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
Uncovering ChatGPT’s Capabilities in Recommender Systems
Dai, S., Shao, N., Zhao, H., Yu, W., Si, Z., Xu, C., Sun, Z., Zhang, X., & Xu, J. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3610646 · Cited by 298
Gallent Torres, C., Zapata González, A., & Ortego Hernando, J. L. (2023) · RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa · doi:10.30827/relieve.v29i2.29134 · Cited by 170
Data-efficient Fine-tuning for LLM-based Recommendation
Lin, X., Wang, W., Li, Y., Yang, S., Feng, F., Wei, Y., & Chua, T. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657807 · Cited by 181
Data-efficient Fine-tuning for LLM-based Recommendation
Lin, X., Wang, W., Li, Y., Yang, S., Feng, F., Wei, Y., & Chua, T.-S. (2024) · arXiv · doi:10.48550/arxiv.2401.17197 · Cited by 165
Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges
Wang, J., Huang, J. X., Tu, X., Wang, J., Huang, A. J., Laskar, M. T. R., & Bhuiyan, A. (2024) · ACM Computing Surveys · doi:10.1145/3648471 · Cited by 156
KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion
Wei, Y., Huang, Q., Zhang, Y., & Kwok, J. T.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.580 · Cited by 143
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.29 · Cited by 70
When Search Engine Services Meet Large Language Models: Visions and Challenges
Xiong, H., Bian, J., Li, Y., Li, X., Du, M., Wang, S., Yin, D., & Helal, S. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3451185 · Cited by 111
Self-citationAccuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81
Evaluating Generative Ad Hoc Information Retrieval
Gienapp, L., Scells, H., Deckers, N., Bevendorff, J., Wang, S., Kiesel, J., Syed, S., Fröbe, M., Zuccon, G., Stein, B., et al. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657849 · Cited by 59
Self-citationEnhancing Recommendation Diversity by Re-ranking with Large Language Models
Carraro, D., & Bridge, D. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3700604 · Cited by 59
Abdollahi, M., Yeganli, S. F., Baharloo, M. M., & Baniasadi, A. (2024) · Electronics · doi:10.3390/electronics14010120 · Cited by 53
Patel, L., Jha, S., Pan, M., Gupta, H., Asawa, P., Guestrin, C., & Zaharia, M. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749685 · Cited by 23
Know where to go: Make LLM a relevant, responsible, and trustworthy searchers
Shi, X. D., Liu, J., Liu, Y., Cheng, Q., & Lu, W. P. (2024) · Decision Support Systems · doi:10.1016/j.dss.2024.114354 · Cited by 51
Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.890 · Cited by 31
Multimodal Misinformation Detection using Large Vision-Language Models
Tahmasebi, S., Müller-Budack, E., & Ewerth, R. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679826 · Cited by 48
Ranked List Truncation for Large Language Model-based Re-Ranking
Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., & de Rijke, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657864 · Cited by 28
RecRanker: Instruction Tuning Large Language Model as Ranker for Top-k Recommendation
Luo, S., He, B., Zhao, H., Shao, W., Qi, Y., Huang, Y., Zhou, A., Yao, Y., Li, Z., Xiao, Y., et al. (2024) · ACM Transactions on Information Systems · doi:10.1145/3705728 · Cited by 69
LLMCDSR: Enhancing Cross-Domain Sequential Recommendation with Large Language Models
Xin, H., Sun, Y., Wang, C., & Xiong, H. (2025) · ACM Transactions on Information Systems · doi:10.1145/3715099 · Cited by 33
Vizniuk, A., Diachenko, G., Laktionov, І., Siwocha, A., Xiao, M., & Smoląg, J. (2025) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2025-0007 · Cited by 36
Can Large Language Models Understand Real-World Complex Instructions?
He, Q., Zeng, J., Huang, W., Chen, L., Xiao, J., He, Q., Zhou, X., Liang, J., & Xiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29777 · Cited by 45
Query Performance Prediction Using Relevance Judgments Generated by Large Language Models
Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., & de Rijke, M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3736402 · Cited by 61
Balog, K., Metzler, D., & Qin, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730348 · Cited by 41
Federated Recommendation via Hybrid Retrieval Augmented Generation
Zeng, H., Yue, Z., Jiang, Q., & Wang, D. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825302 · Cited by 40
Leveraging LLMs for Unsupervised Dense Retriever Ranking
Khramtsova, E., Zhuang, S., Baktashmotlagh, M., & Zuccon, G. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657798 · Cited by 33
Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents
Shi, Z., Gao, S., Yan, L., Feng, Y., Chen, X., Chen, Z., Yin, D., Verberne, S., & Ren, Z. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714825 · Cited by 21
Self-citationA survey on large language models with multilingualism: recent advances and new frontiers
Huang, K., Mo, F., Zhang, X., Li, H., Li, Y., Zhang, Y., yi, W., Mao, Y., Liu, J., Xu, Y., et al. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11534-5 · Cited by 132
Embedding Search for Quranic Texts based on Large Language Models
Alqarni, M. (2024) · The International Arab Journal of Information Technology · doi:10.34028/iajit/21/2/7 · Cited by 17
Ruksakulpiwat, S., Phianhasin, L., Benjasirisan, C., Ding, K., Ajibade, A., Kumar, A., & Stewart, C. (2024) · JMIR mHealth and uHealth · doi:10.2196/51526 · Cited by 17
Multi-granular Adversarial Attacks against Black-box Neural Ranking Models
Liu, Y.-A., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657704 · Cited by 19
Sabbagh, V. B., Lima, C. B. D. C., & Xexeo, G. B. (2024) · SPE Journal · doi:10.2118/223612-pa · Cited by 19
An Investigation of Prompt Variations for Zero-Shot LLM-Based Rankers
Sun, S., Zhuang, S., Wang, S., & Zuccon, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_12 · Cited by 25
Self-citationLLM-enhanced multimodal fusion for cross-domain sequential recommendation
Wu, W., Chen, Z. H., Zhang, W., Song, S., Qiu, X., Huang, X., Ma, F., & Xiao, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132228 · Cited by 29
Self-citationLeveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models
Liu, Q., Wang, B., Wang, N., & Mao, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714554 · Cited by 37
PaRaDe: Passage Ranking using Demonstrations with LLMs
Drozdov, A., Zhuang, H., Dai, Z., Qin, Z., Rahimi, R., Wang, X., Alon, D., Iyyer, M., McCallum, A., Metzler, D., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.950 · Cited by 15
Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers?
Li, M., Zhuang, H., Hui, K., Qin, Z., Lin, J., Jagerman, R., Wang, X., & Bendersky, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657979 · Cited by 23
Song, Z., Hwang, G.-Y., Zhang, X., Huang, S., & Park, B.-K. (2025) · Scientific Reports · doi:10.1038/s41598-025-85715-7 · Cited by 18
CTRL: Connect Collaborative and Language Model for CTR Prediction
Li, X., Chen, B., Hou, L., & Tang, R. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3713080 · Cited by 81
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024) · arXiv · doi:10.48550/arxiv.2407.18940 · Cited by 13
"In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"
Parry, A., Ganguly, D., & Chandra, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657842 · Cited by 15
Large Language Model Powered Agents for Information Retrieval
Zhang, A., Deng, Y., Lin, Y., Chen, X., Wen, J.-R., & Chua, T. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661375 · Cited by 14
Yi, F., Singh, A., & Tao, Z. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000101 · Cited by 13
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
Pradeep, R., Thakur, N., Sharifymoghaddam, S., Zhang, E., Nguyen, R., Campos, D., Craswell, N., & Lin, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_9 · Cited by 46
Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-ranking
Schlatt, F., Fröbe, M., Scells, H., Zhuang, S., Koopman, B., Zuccon, G., Stein, B., Potthast, M., & Hagen, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_31 · Cited by 25
Beyond Utility: Evaluating LLM as Recommender
Jiang, C., Wang, J., Ma, W., Clarke, C. L. A., Wang, S., Wu, C., & Zhang, M. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714759 · Cited by 31
Self-citationGuiding Retrieval Using LLM-Based Listwise Rankers
Rathee, M., MacAvaney, S., & Anand, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_15 · Cited by 16
APEER : Automatic Prompt Engineering Enhances Large Language Model Reranking
Jin, C., Peng, H., Zhao, S., Wang, Z. G., Xu, W., Han, L., Zhao, J., Zhong, K.-W., Rajasekaran, S., & Metaxas, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717574 · Cited by 60
The Convergence of AI/ML and Cybersecurity: Advancing Digital Forensic Techniques
Iyengar, S. S., Nabavirazavi, S., Hariprasad, Y., Prasad, H. B., & Mohan, C. K. (2025) · Signals and communication technology · doi:10.1007/978-3-031-89327-8_4 · Cited by 13
Xiong, Y., Tu, X., & Zhao, W. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104232 · Cited by 13
Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates
Sun, Z., Wan, Y., Li, J., Zhang, H., Jin, Z., Li, G., & Lyu, C. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3695000 · Cited by 10
Is ChatGPT a Good Multi-Party Conversation Solver?
Tan, C.-H., Gu, J.-C., & Ling, Z.-H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.326 · Cited by 16
TWOLAR: A TWO-Step LLM-Augmented Distillation Method for Passage Reranking
Baldelli, D., Jiang, J., Aizawa, A., & Torroni, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_29 · Cited by 10
Enhancing Biomedical Question Answering with Large Language Models
Yang, H., Li, S., & Gonçalves, T. (2024) · Information · doi:10.3390/info15080494 · Cited by 10
Adapting Standard Retrieval Benchmarks to Evaluate Generated Answers
Arabzadeh, N., Bigdeli, A., & Clarke, C. L. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_26 · Cited by 20
RankLLM: A Python Package for Reranking with LLMs
Sharifymoghaddam, S., Pradeep, R., Slavescu, A., Nguyen, R., Xu, A., Chen, Z., Zhang, Y., Chen, Y., Xian, J., & Lin, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730331 · Cited by 20
Self-citationImproving RAG Quality for Large Language Models with Topic-Enhanced Reranking
Ampazis, N. (2024) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-63215-0_6 · Cited by 12
Enhancing ID-based Recommendation with Large Language Models
Chen, L., Gao, C., Du, X., Luo, H., Jin, D., Li, Y., & Wang, M. (2024) · ACM Transactions on Information Systems · doi:10.1145/3704263 · Cited by 12
Self-Calibrated Listwise Reranking with Large Language Models
Ren, R., Wang, Y., Zhou, K., Zhao, W. X., Wang, W., Liu, J., Wen, J.-R., & Chua, T. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714658 · Cited by 12
Criteria-Based LLM Relevance Judgments
Farzi, N., & Dietz, L. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744591 · Cited by 9
Large Language Models for Information Retrieval: Challenges and Chances
Breuer, T., Frihat, S., Fuhr, N., Lewandowski, D., Schaer, P., & Schenkel, R. (2025) · Datenbank-Spektrum · doi:10.1007/s13222-025-00503-x · Cited by 19
Ranking Manipulation for Conversational Search Engines
Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S. (2024) · arXiv · doi:10.48550/arxiv.2406.03589 · Cited by 8
Neural Passage Quality Estimation for Static Pruning
Chang, X., Mishra, D., Macdonald, C., & MacAvaney, S. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657765 · Cited by 9
Lv, Z., Zhan, T., Wang, W., Lin, X., Zhang, S., Zhang, W., Li, J., Kuang, K., & Wu, F. Q. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709335 · Cited by 12
Data Augmentation for Sample Efficient and Robust Document Ranking
Anand, A., Leonhardt, J., Singh, J., Rudra, K., & Anand, A. (2023) · ACM Trans. Inf. Syst. · doi:10.1145/3634911 · Cited by 15
Schlatt, F., Fröbe, M., Scells, H., Zhuang, S., Koopman, B., Zuccon, G., Stein, B., Potthast, M., & Hagen, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_1 · Cited by 16
Data Imputation with Limited Data Redundancy Using Data Lakes
Yang, C., Luo, Y., Cui, C., Fan, J., Chai, C., & Tang, N. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3748191.3748200 · Cited by 13
Robust Neural Information Retrieval: An Adversarial and Out-of-Distribution Perspective
Liu, Y.-A., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3768153 · Cited by 25
Towards Adaptive Personalized Conversational Information Retrieval
Mo, F., Hui, Y., Tian, Y., Tan, Z., Meng, C., Su, Z., Huang, K., & Nie, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761255 · Cited by 7
Accelerating earth science discovery via multi-agent LLM systems
Pantiukhin, D., Shapkin, B., Kuznetsov, I., Jost, A. A., & Koldunov, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1674927 · Cited by 29
Exploring the Impact of Large Language Models on Recommender Systems: An Extensive Review
Vats, A., Raja, R., Jain, V., & Chadha, A. R. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401642 · Cited by 52
Zhang, Y., Huang, B., Yuan, W., Jiang, Z., Peng, L., Chen, S., & Tan‐Soo, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104090 · Cited by 7
LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models
Tang, L., Liu, J., Liu, Z., Yang, X., & Bao, L. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728885 · Cited by 6
Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking
Fang, H., Tao, S., Chen, N., Chang, K.-X., & Sakai, T. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769493 · Cited by 19
Liu, Y.-A., Zhang, R., Guo, J., & de Rijke, M. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703476 · Cited by 6
Rank-Without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models
Zhang, C., Hofstätter, S., Lewis, P., Tang, R., & Lin, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88711-6_15 · Cited by 6
LLM-enhanced representation learning for graph collaborative filtering recommendation models
Mou, D. V., Wei, Z., Ni, L., Song, N., Sun, Y., Chu, W., & Jin, B. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00933-9 · Cited by 6
Leveraging RAG With Transformer for Context-Based Personalized Recommendations
Alamri, F. S., Rehman, A. U., Alghofaily, B., Ahmed, A. R., & Saleem, K. (2025) · IEEE Access · doi:10.1109/access.2025.3574073 · Cited by 9
Unify Graph Learning with Text: Unleashing LLM Potentials for Session Search
Wu, S., Tu, Q., Liu, H., Xu, J., Liu, Z., Zhang, G., Wang, R., Chen, X., & Yan, R. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645574 · Cited by 6
Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System Vulnerability
Chen, N., Liu, J., Fang, H., Luo, Y., Sakai, T., & Wu, X.-M. (2024) · ACM Transactions on Information Systems · doi:10.1145/3708884 · Cited by 12
Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality
Ma, X., Gao, L., Zhuang, S., Zhan, J. S., Callan, J., & Lin, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730135 · Cited by 33
Self-citationReproducing NevIR: Negation in Neural Information Retrieval
van den Elsen, C., Barkhof, F., Nijdam, T., Lupart, S., & Aliannejadi, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730294 · Cited by 10
LLMTrajQuery: an LLM-based generative approach to semantic trajectory queries
Yu, S., Liu, J., Gao, C., Niu, W., Guo, X., Liu, H., & Xu, M. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2569745 · Cited by 5
Chopra, A., & Azam, R. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-62495-7_12 · Cited by 5
Enhancing Dataset Search with Compact Data Snippets
Chen, Q., Chen, J., Zhou, X., & Cheng, G. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657837 · Cited by 7
Ruan, Q., Shi, D., & Bauernhansl, T. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103538 · Cited by 5
LLM-BL: Large Language Models are Zero-Shot Rankers for Bug Localization
Li, Z., Jiang, Z., Huang, Q., & Gu, Q. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00064 · Cited by 4
Applying Large Language Model For Relevance Search In Tencent
Ye, D., Liu, J., Hu, J., Fan, J., Tian, B., Liang, H., & Ma, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737193 · Cited by 7
Abdallah, A., Abdalla, M. I., Piryani, B., Mozafari, J., Ali, M., & Jatowt, A. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761484 · Cited by 6
How Do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective
Liu, Q., Duan, H., Mao, J., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3774942 · Cited by 13
Liu, Y.-A., Zhang, R., Guo, J., & de Rijke, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661380 · Cited by 4
Neural Lexical Search with Learned Sparse Retrieval
Yates, A., Lassance, C., MacAvaney, S., Nguyen, T. D.-M., & Lei, Y. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698441 · Cited by 5
Rocchietti, G., Rulli, C., Nardini, F. M., Muntean, C. I., Perego, R., & Frieder, O. (2025) · IEEE Access · doi:10.1109/access.2025.3529741 · Cited by 6
Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-Agent LLMs
Li, X., Su, L., Jia, P., Cheng, S., Wang, J., Yin, D., & Zhao, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3749099 · Cited by 39
Self-citationContextual Interaction for Argument Post Quality Assessment
Wang, Y., Chen, X., He, B., & Sun, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.645 · Cited by 4
Rewriting Conversational Utterances with Instructed Large Language Models
Galimzhanova, E., Muntean, C. I., Nardini, F. M., Perego, R., & Rocchietti, G. (2023) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat59888.2023.00014 · Cited by 6
Metadata
- Authors
- Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., & Ren, Z.
- Affiliations
- Shandong University (CN)
- Baidu (China) (CN)
- Leiden University (NL)
- Venue
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Published
- 2023
- Pages
- 14918-14937
- OpenAlex
- W4389523765
- Cited by
- 270
- References
- 39
- Citations by year
- 2026: 94 · 2025: 104 · 2024: 56 · 2023: 16
- FWCI
- 69.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Leiden Repository (Leiden University) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Shandong University
- National Natural Science Foundation of China (61972234, 62102234, 62272274, 62202271, 62072279)
- China Scholarship Council
- Natural Science Foundation of Shandong Province (ZR2021QF129)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- large language models, ChatGPT, GPT-4, re-ranking, information retrieval, passage ranking, relevance ranking, contaminated data, knowledge distillation, BEIR benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
