CoChat

MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

Open access
Craswell, Nick, Mitra, Bhaskar, Yilmaz, Emine, Campos, Daniel, Lin, Jimmy · 2021 · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
Cited by 55

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3404835.3462804

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 78

  1. James Allan, Donna Harman, Evangelos Kanoulas, Dan Li, Christophe Van Gysel, and Ellen M Voorhees. 2017. TREC 2017 Common Core Track Overview.. In TREC.
  2. 10.1145/1645953.1646031 · DOI · Paper page
  3. Michael Bendersky, Honglei Zhuang, Ji Ma, Shuguang Han, Keith Hall, and Ryan McDonald. 2020. RRF102: Meeting the TREC-COVID challenge with a 100+ runs ensemble. arXiv preprint arXiv:2010.00200 (2020). · arXiv
  4. 10.1145/345508.345543 · DOI · Paper page
  5. 10.1145/2766462.2767812 · DOI
  6. 10.1145/1046456.1046470 · doi:10.1145/1046456.1046470
  7. Jimmy Chen and William Hersh. 2020. A Comparative Analysis of System Features Used in the TREC-COVID Information Retrieval Challenge. medRxiv (2020).
  8. Charles LA Clarke, Nick Craswell, and Ian Soboroff. 2004. Overview of the TREC 2004 Terabyte Track.. In TREC, Vol. 4. 74.
  9. Nick Craswell. 2009. Mean Reciprocal Rank. Encyclopedia of database systems, Vol. 1703 (2009).
  10. 10.1145/3159652.3159659 · DOI · Paper page
  11. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020 a. TREC CAsT 2019: The conversational assistance track overview. arXiv preprint arXiv:2003.13624 (2020). · arXiv
  12. 10.1145/3397271.3401206 · DOI · Paper page
  13. 10.1145/3077136.3080832 · DOI · Paper page
  14. Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. 2017. TREC complex answer retrieval overview. In Proceedings of TREC.
  15. Marco Ferrante, Nicola Ferro, and Norbert Fuhr. 2021. Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales. arXiv preprint arXiv:2101.02668 (2021). · arXiv
  16. 10.1145/3121050.3121058 · DOI
  17. 10.1109/TKDE.2018.2840708 · DOI
  18. Rafael Ferreira, Mariana Leite, David Semedo, and Joao Magalhaes. 2021. Open-Domain Conversational Search Assistant with Transformers. arXiv preprint arXiv:2101.08197 (2021). · arXiv
  19. Norbert Fuhr. [n.d.]. SIGIR Keynote: Proof By Experimentation? Towards Better IR Research. ( [n. d.]).
  20. 10.1145/3190580.3190586 · DOI
  21. 10.1145/3397271.3402426 · doi:10.1145/3397271.3402426
  22. 10.18653/v1/2020.emnlp-main.342 · DOI
  23. Ehsan Hamzei, Haonan Li, Maria Vasardani, Timothy Baldwin, Stephan Winter, and Martin Tomko. 2019. Place questions and human-generated answers: A data analysis approach. In International Conference on Geographic Information Science. Springer, 3--19.
  24. 10.1145/160688.160692 · DOI · Paper page
  25. 10.1561/9781680831627 · doi:10.1561/9781680831627
  26. Po-Sen Huang Xiaodong He Jianfeng Gao Li Deng Alex Acero and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. ACM 2333--2338. · DOI · Paper page
  27. 10.1145/160688.160758 · DOI · Paper page
  28. Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, and Wei Wang. 2020. Long Document Ranking with Query-Directed Sparse Transformer. arXiv preprint arXiv:2010.12683 (2020). · arXiv
  29. David H Krantz, Patrick Suppes, and R Duncan Luce. 2006. Foundations of Measurement: Additive and polynomial representations. Vol. 1. Courier Corporation.
  30. 10.18653/v1/2020.findings-emnlp.354 · DOI
  31. Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. PARADE: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093 (2020). · arXiv
  32. 10.1145/3308774.3308781 · DOI · Paper page
  33. 10.1145/3458553.3458563 · DOI
  34. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained Transformers for Text Ranking: BERT and Beyond. arxiv: 2010.06467 [cs.IR] · arXiv
  35. 10.18653/v1/2020.findings-emnlp.216 · DOI
  36. Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2020. PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval. arXiv preprint arXiv:2010.10137 (2020). · arXiv
  37. Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2020 a. SLEDGE: A simple yet effective baseline for coronavirus scientific knowledge search. arXiv preprint arXiv:2005.02365 (2020). · arXiv
  38. Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2020 b. SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search. arXiv preprint arXiv:2010.05987 (2020). · arXiv
  39. 10.1145/3397271.3401268 · DOI
  40. Bhaskar Mitra Fernando Diaz and Nick Craswell. 2017. Learning to Match Using Local and Distributed Representations of Text for Web Search. 1291--1299. · DOI · Paper page
  41. Vincent Nguyen, Maciek Rybinski, Sarvnaz Karimi, and Zhenchang Xing. 2020. Pandemic Literature Search: Finding Information on COVID-19. In Proceedings of the The 18th Annual Workshop of the Australasian Language Technology Association. 92--97.
  42. Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020 a. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020). · arXiv
  43. 10.18653/v1 · DOI
  44. 10.1145/1835449.1835560 · DOI
  45. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
  46. Pengjie Ren, Zhumin Chen, Zhaochun Ren, Evangelos Kanoulas, Christof Monz, and Maarten de Rijke. 2020. Conversations with search engines. arXiv preprint arXiv:2004.14162 (2020). · arXiv
  47. 10.1093/jamia/ocaa091 · DOI · Paper page
  48. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. Nist Special Publication Sp, Vol. 109 (1995), 109.
  49. 10.1145/3209978.3210127 · DOI · Paper page
  50. Yastil Rughbeer, Anban W Pillay, and Edgar Jembere. 2021. Dataset Selection for Transfer Learning in Information Retrieval. In Southern African Conference for Artificial Intelligence Research. Springer, 53--65.
  51. Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision Vol. 115 3 (2015) 211--252. · DOI · Paper page
  52. 10.1145/3451964.3451976 · DOI
  53. Tetsuya Sakai, Sijie Tao, Zhaohao Zeng, Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu, Maria Maistro, Zhicheng Dou, Nicola Ferro, et al. 2020. Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task. Proceedings of NTCIR-15. to appear (2020).
  54. 10.1145/1076034.1076064 · DOI · Paper page
  55. Vasileios Stamatis, Leif Azzopardi, and Alan Wilson. 2019. VES Team at TREC Conversational Assistance Track (CAsT) 2019.. In TREC.
  56. Si Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang, Jie Bao, Zhiyuan Liu, and Paul Bennett. 2020. Meta Adaptive Neural Ranking with Contrastive Synthetic Supervision. arXiv preprint arXiv:2012.14862 (2020). · arXiv
  57. 10.1145/2983323.2983702 · DOI · Paper page
  58. Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020. TREC-COVID: constructing a pandemic information retrieval test collection. arXiv preprint arXiv:2005.04474 (2020). · arXiv
  59. Ellen M Voorhees. 2004. Overview of the TREC 2004 Robust Track. In Trec.
  60. 10.1145/3397271.3402427 · doi:10.1145/3397271.3402427
  61. Ellen M Voorhees, Donna K Harman, et al. 2005. TREC: Experiment and evaluation in information retrieval. Vol. 63. MIT press Cambridge, MA.
  62. Nikos Voskarides Dan Li Andreas Panteli and Pengjie Ren. 2019. ILPS at TREC 2019 Conversational Assistant Track. In TREC. · doi:10.6028/nist.sp.1250.cast-uva.ilps
  63. 10.1145/3397271.3401130 · DOI
  64. Disen Wang and Hui Fang. 2019. Exploring Query Reformulation for Conversational Information Seeking.. In TREC. · doi:10.6028/nist.sp.1250.cast-udel_fang
  65. Lucy Lu Wang and Kyle Lo. 2020. Text mining approaches for dealing with the rapidly expanding literature on COVID-19. Briefings in Bioinformatics (2020).
  66. Lucy Lu Wang Kyle Lo Yoganand Chandrasekhar Russell Reas Jiangjiang Yang Darrin Eide Kathryn Funk Rodney Kinney Ziyang Liu William Merrill et al. 2020. Cord-19: The covid-19 open research dataset. ArXiv (2020).
  67. 10.1145/3077136.3080809 · DOI · Paper page
  68. Chenyan Xiong, Zhenghao Liu, Si Sun, Zhuyun Dai, Kaitao Zhang, Shi Yu, Zhiyuan Liu, Hoifung Poon, Jianfeng Gao, and Paul Bennett. 2020. CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search. arXiv preprint arXiv:2011.01580 (2020). · arXiv
  69. 10.18653/v1/D19-1352 · DOI · Paper page
  70. 10.1145/3397271.3401323 · DOI · Paper page
  71. 10.1145/3159652.3159730 · DOI
  72. Edwin Zhang, Nikhil Gupta, Rodrigo Nogueira, Kyunghyun Cho, and Jimmy Lin. 2020 a. Rapidly deploying a neural search engine for the covid-19 open research dataset: Preliminary thoughts and lessons learned. arXiv preprint arXiv:2004.05125 (2020). · arXiv
  73. Edwin Zhang, Nikhil Gupta, Raphael Tang, Xiao Han, Ronak Pradeep, Kuang Lu, Yue Zhang, Rodrigo Nogueira, Kyunghyun Cho, Hui Fang, et al. 2020 b. Covidex: Neural ranking models and keyword search infrastructure for the covid-19 open research dataset. arXiv preprint arXiv:2007.07846 (2020). · arXiv
  74. 10.1145/3366423.3380131 · DOI
  75. 10.18653/v1/2020.sustainlp-1.14 · DOI
  76. Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020. BERT-QE: contextualized query expansion for document re-ranking. arXiv preprint arXiv:2009.07258 (2020). · arXiv
  77. Mu Zhu. 2004. Recall, Precision and Average Precision. Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, Vol. 2 (2004), 30.
  78. 10.1145/281250.281256 · DOI · Paper page

Cited by

See all 55

Metadata

Authors
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Lin, J.
Venue
Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2021-07-11
Pages
1566-1576
OpenAlex
W3152624702
Cited by
55
References
11
FWCI
8.53
Type
conference paper
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.