CoChat

LitSearch: A Retrieval Benchmark for Scientific Literature Search

Ajith, Anirudh, Xia, Mengzhou, Chevalier, Alexis, Goyal, Tanya, Chen, Danqi, Gao, Tianyu · 2024 · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Cited by 23

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

Literature search questions, such as "Where can I find research on the evaluation of consistency in generated summaries?"pose significant challenges for modern search engines and retrieval systems.These questions often require a deep understanding of research concepts and the ability to reason across entire articles.In this work, we introduce LitSearch, a retrieval benchmark comprising 597 realistic literature search queries about recent ML and NLP papers.Lit-Search is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and (2) questions manually written by authors about their recently published papers.All LitSearch questions were manually examined or edited by experts to ensure high quality.We extensively benchmark state-ofthe-art retrieval models and also evaluate two LLM-based reranking pipelines.We find a significant performance gap between BM25 and state-of-the-art dense retrievers, with a 24.8% absolute difference in recall@5.The LLMbased reranking strategies further improve the best-performing dense retriever by 4.4%.Additionally, commercial search engines and research tools like Google Search perform poorly on LitSearch, lagging behind the best dense retriever by up to 32 recall points.Taken together, these results show that LitSearch is an informative new testbed for retrieval systems while catering to a real-world use case.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-main.840

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Examples of inline-citation and author-written questions from LitSearch. These questions are often challenging and require a deep understanding of the target papers to answer correctly.
Figure 1 Examples of inline-citation and author-written questions from LitSearch. These questions are often challenging and require a deep understanding of the target papers to answer correctly.CC-BY · From the PDF
Figure 2 The pipeline for generating inline-citation questions. We first sample a citation mention and prompt GPT-4 to generate a question. Next, we filter questions based on word overlap with the target paper title and perform manual inspections to annotate their specificity and quality (see rubrics in Table 1).
Figure 2 The pipeline for generating inline-citation questions. We first sample a citation mention and prompt GPT-4 to generate a question. Next, we filter questions based on word overlap with the target paper title and perform manual inspections to annotate their specificity and quality (see rubrics in Table 1).CC-BY · From the PDF
Figure 3 We demonstrate detailed retrieval results using BM25, E5 and GritLM up to k = 50 . Additionally, we show the effect of applying GPT-4o reranking over GritLM retrieval results.
Figure 3 We demonstrate detailed retrieval results using BM25, E5 and GritLM up to k = 50 . Additionally, we show the effect of applying GPT-4o reranking over GritLM retrieval results.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 33

  1. BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language mod- els are secretly powerful text encoders. Preprint, arXiv:2404.05961. · arXiv
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
  3. Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018. Content-based citation recommendation. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pages 238-251, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Process- ing Systems (NeurIPS).
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language under- standing. In North American Chapter of the Associa- tion for Computational Linguistics (NAACL).
  6. Michael Färber and Adam Jatowt. 2020. Citation rec- ommendation: approaches and datasets. Int. J. Digit. Libr., 21(4):375-405. · DOI · Paper page
  7. Nianlong Gu, Yingqiang Gao, and Richard H. R. Hahn- loser. 2022. Local citation recommendation with hierarchical-attention text encoder and scibert-based reranking. In Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Pro- ceedings, Part I, page 274-288, Berlin, Heidelberg. Springer-Verlag. · DOI
  8. Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles. 2010. Context-aware citation recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, page 421-430, New York, NY, USA. Association for Computing Machin- ery. · DOI · Paper page
  9. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
  10. Chanwoo Jeong, Sion Jang, Eunjeong Park, and Sungchul Choi. 2020. A context-aware citation rec- ommendation model with bert and graph convolu- tional networks. Scientometrics, 124(3):1907-1922. · DOI · Paper page
  11. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics. · DOI · Paper page
  12. O. Khattab and Matei A. Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466. · DOI · Paper page
  14. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. Preprint, arXiv:2405.17428. · arXiv
  15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Association for Com- putational Linguistics (ACL), pages 6086-6096. · DOI · Paper page
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. · arXiv
  17. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Association for Computational Linguistics (ACL), pages 4969-4983. · DOI · Paper page
  18. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156. · arXiv
  19. Zoran Medić and Jan Snajder. 2020. Improved local citation recommendation based on context enhanced with global information. In Proceedings of the First Workshop on Scholarly Document Processing, pages 97-103, Online. Association for Computational Lin- guistics. · DOI
  20. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational in- struction tuning. arXiv preprint arXiv:2402.09906. · arXiv
  21. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Empirical Methods in Natural Language Processing (EMNLP), pages 9844-9855.
  22. OpenAI. 2023. GPT-4 Technical Report. Preprint, arXiv:2303.08774. · arXiv
  23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. · DOI · Paper page
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Empirical Methods in Natural Lan- guage Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  25. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends® in Information Re- trieval, 3(4):333-389.
  26. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text em- beddings. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 1102-1121, Toronto, Canada. Association for Computational Lin- guistics. · DOI · Paper page
  27. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023a. Is chatgpt good at search? investigating large language models as re-ranking agent. ArXiv, abs/2304.09542. · arXiv
  28. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023b. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Empirical Methods in Natural Language Processing (EMNLP), pages 14918-14937. · DOI · Paper page
  29. Michael Tang, Shunyu Yao, John Yang, and Karthik Narasimhan. 2023. Referral augmenta- tion for zero-shot information retrieval. Preprint, arXiv:2305.15098. · arXiv
  30. Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  31. Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '00, page 200-207, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  32. Liang Wang, Nan Yang, Xiaolong Huang, Binx- ing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. ArXiv, abs/2212.03533. · arXiv
  33. Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022. Grit: A generative region-to-text transformer for ob- ject understanding. Preprint, arXiv:2212.00280. · arXiv

Cited by

See all 23

Metadata

Authors
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T.
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Published
2024
Pages
15068-15083
Cited by
23
References
8

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: LitSearch: A Retrieval Benchmark for Scientific Literature Search. https://app.cochat.ai/papers/10.18653/v1/2024.emnlp-main.840
The paper
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024). LitSearch: A Retrieval Benchmark for Scientific Literature Search. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2024.emnlp-main.840

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.