CoChat
Sign Up
PreprintCited by 5Information Retrieval and Search BehaviorTopic ModelingAdvanced Image and Video Retrieval Techniques

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

In Defense of Cross-Encoders for Zero-Shot Retrieval

Open access
Rosa, Guilherme, Bonifacio, Luiz, Jeronymo, Vitor, Abonizio, Hugo, Fadaee, Marzieh, Lotufo, Roberto, Nogueira, Rodrigo·2022·arXiv
DOI 10.48550/arxiv.2212.06121arXiv 2212.06121
Open this paper in CoChat — free accountOpen PDF

Abstract

Bi-encoders and cross-encoders are widely used in many state-of-the-art retrieval pipelines. In this work we study the generalization ability of these two types of architectures on a wide range of parameter count on both in-domain and out-of-domain scenarios. We find that the number of parameters and early query-document interactions of cross-encoders play a significant role in the generalization ability of retrieval models. Our experiments show that increasing model size results in marginal gains on in-domain test sets, but much larger gains in new domains never seen during fine-tuning. Furthermore, we show that cross-encoders largely outperform bi-encoders of similar size in several tasks. In the BEIR benchmark, our largest cross-encoder surpasses a state-of-the-art bi-encoder by more than 4 average points. Finally, we show that using bi-encoders as first-stage retrievers provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks. The code is available at https://github.com/guilhermemr04/scaling-zero-shot-retrieval.git

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2212.06121

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 53

  1. [1]1. Bai, Y., Li, X., Wang, G., Zhang, C., Shang, L., Xu, J., Wang, Z., Wang, F., Liu, Q.: Sparterm: Learning term-based sparse representation for fast text retrieval. arXiv preprint arXiv:2010.00768 (2020) · arXiv
  2. [2]2. Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenber, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2018) · arXiv
  3. [3]3. Bonifacio, L., Abonizio, H., Fadaee, M., Nogueira, R.: Inpars: Unsupervised dataset generation for information retrieval. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2387-2392 (2022)
  4. [4]4. Boytsov, L., Nyberg, E.: Flexible retrieval with nmslib and flexneuart. In: Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS). pp. 32-43 (2020)
  5. [5]5. Craswell, N., Mitra, B., Yilmaz, E., Campos, D.: Overview of the trec 2020 deep learning track (2021). https://doi.org/10.48550/ARXIV.2102.07662 , https:// arxiv.org/abs/2102.07662 · DOI · Paper page
  6. [6]6. Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Lin, J.: Overview of the trec 2021 deep learning track. In: Text REtrieval Conference (TREC). TREC (May 2022), https://www.microsoft.com/en-us/research/publication/ overview-of-the-trec-2021-deep-learning-track/
  7. [7]7. Dai, Z., Callan, J.: Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv:1910.10687 (2019) · arXiv
  8. [9]9. Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 (2022) · arXiv
  9. [10]10. Dettmers, T., Lewis, M., Shleifer, S., Zettlemoyer, L.: 8-bit optimizers via blockwise quantization. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=shpkpVXzo3h
  10. [11]11. Formal, T., Lassance, C., Piwowarski, B., Clinchant, S.: Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021) · arXiv
  11. [12]12. Formal, T., Lassance, C., Piwowarski, B., Clinchant, S.: From distillation to hard negative sampling: Making sparse neural ir models more effective. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 2353-2359. SIGIR '22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3477495. 3531857 , https://doi.org/10.1145/3477495.3531857 · DOI
  12. [13]13. Formal, T., Piwowarski, B., Clinchant, S.: Splade: Sparse lexical and expansion model for first stage ranking. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2288-2292 (2021)
  13. [14]14. Gao, L., Callan, J.: Unsupervised corpus aware language model pre-training for dense passage retrieval. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2843-2853 (2022)
  14. [15]15. Gao, L., Callan, J.: Unsupervised corpus aware language model pre-training for dense passage retrieval. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 28432853. Association for Computational Linguistics, Dublin, Ireland (May 2022), https://aclanthology.org/2022.acl-long.203
  15. [16]16. Gao, L., Dai, Z., Callan, J.: Rethink training of bert rerankers in multi-stage retrieval pipeline. arXiv preprint arXiv:2101.08751 (2021) · arXiv
  16. [17]17. Gupta, P., MacAvaney, S.: On survivorship bias in ms marco. arXiv preprint arXiv:2204.12852 (2022) · arXiv
  17. [18]18. Hofstätter, S., Khattab, O., Althammer, S., Sertkan, M., Hanbury, A.: Introducing neural bag of whole-words with colberter: Contextualized late interactions using enhanced reduction. arXiv preprint arXiv:2203.13088 (2022) · arXiv
  18. [19]19. Hofstätter, S., Lin, S.C., Yang, J.H., Lin, J., Hanbury, A.: Efficiently teaching an effective dense retriever with balanced topic aware sampling. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 113-122 (2021)
  19. [21]21. Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning (2021). https://doi.org/10.48550/ARXIV.2112.09118 , https://arxiv.org/ abs/2112.09118 · DOI · Paper page
  20. [22]22. Li, C., Yates, A., MacAvaney, S., He, B., Sun, Y.: Parade: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093 (2020) · arXiv
  21. [24]24. Lin, J., Ma, X.: A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807 (2021) · arXiv
  22. [25]25. Lin, J., Ma, X., Lin, S.C., Yang, J.H., Pradeep, R., Nogueira, R.: Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations. arXiv preprint arXiv:2102.10073 (2021) · arXiv
  23. [26]26. Lu, S., He, D., Xiong, C., Ke, G., Malik, W., Dou, Z., Bennett, P., Liu, T.Y., Overwijk, A.: Less is more: Pretrain a strong Siamese encoder for dense text retrieval using a weak decoder. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 2780-2791. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.220 , https://aclanthology. org/2021.emnlp-main.220 · DOI · Paper page
  24. [27]27. MacAvaney, S., Yates, A., Cohan, A., Goharian, N.: Cedr: Contextualized embeddings for document ranking. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1101-1104 (2019)
  25. [28]28. Mallia, A., Khattab, O., Suel, T., Tonellotto, N.: Learning passage impacts for inverted indexes. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1723-1727 (2021)
  26. [29]29. Menon, A., Jayasumana, S., Rawat, A.S., Kim, S., Reddi, S., Kumar, S.: In defense of dual-encoders for neural ranking. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 15376-15400. PMLR (17-23 Jul 2022), https://proceedings. mlr.press/v162/menon22a.html
  27. [30]30. Muennighoff, N.: Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904 (2022) · arXiv
  28. [31]31. Ni, J., Qu, C., Lu, J., Dai, Z., Ábrego, G.H., Ma, J., Zhao, V.Y., Luan, Y., Hall, K.B., Chang, M.W., Yang, Y.: Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899 (2021) · arXiv
  29. [32]32. Nogueira, R., Cho, K.: Passage re-ranking with bert. arXiv preprint arXiv:1901.04085 (2019) · arXiv
  30. [33]33. Nogueira, R., Jiang, Z., Pradeep, R., Lin, J.: Document ranking with a pretrained sequence-to-sequence model. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. pp. 708-718 (2020)
  31. [34]34. Nogueira, R., Lin, J.: From doc2query to doctttttquery. Online preprint 6 (2019)
  32. [35]35. Nogueira, R., Yang, W., Lin, J., Cho, K.: Document expansion by query prediction. arXiv preprint arXiv:1904.08375 (2019) · arXiv
  33. [38]38. Qiao, Y., Chen, H., Lai, Y., Wang, J., Liu, T., Ye, X., Fang, R., Gao, P., Xie, W., Xie, G.T.: Pash at trec 2021 deep learning track: Generative enhanced model for multi-stage ranking. ArXiv abs/2205.11245 (2022) · arXiv
  34. [39]39. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified textto-text transformer. Journal of Machine Learning Research 21 (140), 1-67 (2020), http://jmlr.org/papers/v21/20-074.html
  35. [40]40. Ren, R., Qu, Y., Liu, J., Zhao, W.X., Wu, Q., Ding, Y., Wu, H., Wang, H., Wen, J.R.: A thorough examination on zero-shot dense retrieval. arXiv preprint arXiv:2204.12755 (2022) · arXiv
  36. [41]41. Roberts, K., Demner-Fushman, D., Voorhees, E., Hersh, W., Bedrick, S., Lazar, A.J., Pant, S.: Overview of the trec 2019 precision medicine track. The ... text REtrieval conference : TREC. Text REtrieval Conference 26 (2019)
  37. [42]42. Rosa, G.M., Rodrigues, R.C., Lotufo, R., Nogueira, R.: To tune or not to tune? zero-shot models for legal case entailment. ICAIL'21, Eighteenth International Conference on Artificial Intelligence and Law, June 21-25, 2021, São Paulo, Brazil (2021)
  38. [43]43. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., Zaharia, M.: ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 37153734. Association for Computational Linguistics, Seattle, United States (Jul 2022). https://doi.org/10.18653/v1/2022.naacl-main.272 , https://aclanthology. org/2022.naacl-main.272 · DOI · Paper page
  39. [44]44. Tay, Y., Dehghani, M., Abnar, S., Chung, H.W., Fedus, W., Rao, J., Narang, S., Tran, V.Q., Yogatama, D., Metzler, D.: Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551 (2022) · arXiv
  40. [45]45. Thakur, N., Reimers, N., Lin, J.: Domain adaptation for memory-efficient dense retrieval. arXiv preprint arXiv:2205.11498 (2022) · arXiv
  41. [46]46. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I.: Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 (2021) · arXiv
  42. [47]47. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS (2017)
  43. [48]48. Voorhees, E.M.: Overview of the trec 2004 robust track. Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004, Gaithersburg, Maryland, November 16-19, 2004 (2004)
  44. [49]49. Wang, K., Thakur, N., Reimers, N., Gurevych, I.: Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. arXiv preprint arXiv:2112.07577 (2021) · arXiv
  45. [50]50. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957 (2020) · arXiv
  46. [51]51. Wang, Y.E., Wei, G.Y., Brooks, D.: Benchmarking tpu, gpu, and cpu platforms for deep learning. arXiv preprint arXiv:1907.10701 (2019) · arXiv
  47. [52]52. Wu, X., Ma, G., Lin, M., Lin, Z., Wang, Z., Hu, S.: Contextual mask auto-encoder for dense passage retrieval. arXiv preprint arXiv:2208.07670 (2022) · arXiv
  48. [53]53. Xin, J., Xiong, C., Srinivasan, A., Sharma, A., Jose, D., Bennett, P.N.: Zeroshot dense retrieval with momentum adversarial domain invariant representations. arXiv preprint arXiv:2110.07581 (2021) · arXiv
  49. [54]54. Yoo, J., Perlin, K., Kamalakara, S.R., Araújo, J.G.: Scalable training of language models using jax pjit and tpuv4. arXiv preprint arXiv:2204.06514 (2022) · arXiv
  50. [55]55. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al.: Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 (2022) · arXiv
  51. [56]56. Zhan, J., Xie, X., Mao, J., Liu, Y., Zhang, M., Ma, S.: Evaluating extrapolation performance of dense retrieval. arXiv preprint arXiv:2204.11447 (2022) · arXiv
  52. [57]57. Zhang, E., Gupta, N., Nogueira, R., Cho, K., Lin, J.: Rapidly deploying a neural search engine for the covid-19 open research dataset. In: Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020 (2020)
  53. [58]58. Zhang, Y., Long, D., Xu, G., Xie, P.: Hlatr: Enhance multi-stage text retrieval with hybrid list aware transformer reranking. arXiv preprint arXiv:2205.10569 (2022) · arXiv

Cited by

Sorted by citations

Metadata

Authors
Rosa, G., Bonifacio, L., Jeronymo, V., Abonizio, H., Fadaee, M., Lotufo, R., & Nogueira, R.
Affiliations
Universidade Estadual de Campinas (UNICAMP) (BR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2212.06121
arXiv
2212.06121
OpenAlex
W4311432385
Cited by
5
References
1
Citations by year
2025: 2 · 2024: 1 · 2023: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
cross-encoder, bi-encoder, zero-shot retrieval, generalization, out-of-domain generalization, model size, BM25, BEIR benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review