MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“The adoption of MS MARCO in so many transfer learning settings is encouraging, and while it may be premature to draw parallels between its impact on the IR community and what ImageNet [20, 64] did for computer vision research, the current trends definitely bode well for MS MARCO’s potential role in the future of IR re…”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841 · cited 1×
“Commonly used information retrieval metrics, including the ones we employed in our leaderboard evaluation such as NDCG [36] and MRR [12], have recently been criticized by Ferrante et al.”
MethodMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Self-citationNogueira, R., & Cho, K. (2019) · arXiv · doi:10.48550/arxiv.1901.04085 · Cited by 615 · cited 2×
“On left panel in Figure 1 for the passage leaderboard, the large jump in the SOTA in January 2019 represents the work of Nogueira and Cho [54], which is the first known application of pretrained transformers to a ranking task.”
BackgroundDeep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350 · cited 1×
“on one hand, and a flurry of new research and development of IR systems specializing on this task [9, 69, 80] on the other.”
BackgroundOverview of the TREC 2020 Deep Learning Track
Craswell, N., Mitra, B., Yilmaz, E., & Campos, D. (2021) · arXiv · doi:10.48550/arxiv.2102.07662 · Cited by 107 · cited 2×
“[13, 14] have reported, several pretraining-based deep models finetuned on the MS MARCO training data achieve significant improvements over traditional IR methods in this setting.”
BackgroundSelf-citationOverview of the TREC 2019 deep learning track
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Voorhees, E. M. (2020) · arXiv · doi:10.48550/arxiv.2003.07820 · Cited by 68
Self-citationTREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E. M., & Soboroff, I. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463249 · Cited by 58 · cited 1×
“I R ] 9 M a y 2021 leader-boards and the TREC Deep Learning Track [13–15].”
BackgroundSelf-citation
Research works printed but not linked: 78
- James Allan, Donna Harman, Evangelos Kanoulas, Dan Li, Christophe Van Gysel, and Ellen M Voorhees. 2017. TREC 2017 Common Core Track Overview.. In TREC.
- 10.1145/1645953.1646031 · DOI · Paper page
- Michael Bendersky, Honglei Zhuang, Ji Ma, Shuguang Han, Keith Hall, and Ryan McDonald. 2020. RRF102: Meeting the TREC-COVID challenge with a 100+ runs ensemble. arXiv preprint arXiv:2010.00200 (2020). · arXiv
- 10.1145/345508.345543 · DOI · Paper page
- 10.1145/2766462.2767812 · DOI
- 10.1145/1046456.1046470 · doi:10.1145/1046456.1046470
- Jimmy Chen and William Hersh. 2020. A Comparative Analysis of System Features Used in the TREC-COVID Information Retrieval Challenge. medRxiv (2020).
- Charles LA Clarke, Nick Craswell, and Ian Soboroff. 2004. Overview of the TREC 2004 Terabyte Track.. In TREC, Vol. 4. 74.
- Nick Craswell. 2009. Mean Reciprocal Rank. Encyclopedia of database systems, Vol. 1703 (2009).
- 10.1145/3159652.3159659 · DOI · Paper page
- Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020 a. TREC CAsT 2019: The conversational assistance track overview. arXiv preprint arXiv:2003.13624 (2020). · arXiv
- 10.1145/3397271.3401206 · DOI · Paper page
- 10.1145/3077136.3080832 · DOI · Paper page
- Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. 2017. TREC complex answer retrieval overview. In Proceedings of TREC.
- Marco Ferrante, Nicola Ferro, and Norbert Fuhr. 2021. Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales. arXiv preprint arXiv:2101.02668 (2021). · arXiv
- 10.1145/3121050.3121058 · DOI
- 10.1109/TKDE.2018.2840708 · DOI
- Rafael Ferreira, Mariana Leite, David Semedo, and Joao Magalhaes. 2021. Open-Domain Conversational Search Assistant with Transformers. arXiv preprint arXiv:2101.08197 (2021). · arXiv
- Norbert Fuhr. [n.d.]. SIGIR Keynote: Proof By Experimentation? Towards Better IR Research. ( [n. d.]).
- 10.1145/3190580.3190586 · DOI
- 10.1145/3397271.3402426 · doi:10.1145/3397271.3402426
- 10.18653/v1/2020.emnlp-main.342 · DOI
- Ehsan Hamzei, Haonan Li, Maria Vasardani, Timothy Baldwin, Stephan Winter, and Martin Tomko. 2019. Place questions and human-generated answers: A data analysis approach. In International Conference on Geographic Information Science. Springer, 3--19.
- 10.1145/160688.160692 · DOI · Paper page
- 10.1561/9781680831627 · doi:10.1561/9781680831627
- Po-Sen Huang Xiaodong He Jianfeng Gao Li Deng Alex Acero and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. ACM 2333--2338. · DOI · Paper page
- 10.1145/160688.160758 · DOI · Paper page
- Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, and Wei Wang. 2020. Long Document Ranking with Query-Directed Sparse Transformer. arXiv preprint arXiv:2010.12683 (2020). · arXiv
- David H Krantz, Patrick Suppes, and R Duncan Luce. 2006. Foundations of Measurement: Additive and polynomial representations. Vol. 1. Courier Corporation.
- 10.18653/v1/2020.findings-emnlp.354 · DOI
- Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. PARADE: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093 (2020). · arXiv
- 10.1145/3308774.3308781 · DOI · Paper page
- 10.1145/3458553.3458563 · DOI
- Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained Transformers for Text Ranking: BERT and Beyond. arxiv: 2010.06467 [cs.IR] · arXiv
- 10.18653/v1/2020.findings-emnlp.216 · DOI
- Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2020. PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval. arXiv preprint arXiv:2010.10137 (2020). · arXiv
- Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2020 a. SLEDGE: A simple yet effective baseline for coronavirus scientific knowledge search. arXiv preprint arXiv:2005.02365 (2020). · arXiv
- Sean MacAvaney, Arman Cohan, and Nazli Goharian. 2020 b. SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search. arXiv preprint arXiv:2010.05987 (2020). · arXiv
- 10.1145/3397271.3401268 · DOI
- Bhaskar Mitra Fernando Diaz and Nick Craswell. 2017. Learning to Match Using Local and Distributed Representations of Text for Web Search. 1291--1299. · DOI · Paper page
- Vincent Nguyen, Maciek Rybinski, Sarvnaz Karimi, and Zhenchang Xing. 2020. Pandemic Literature Search: Finding Information on COVID-19. In Proceedings of the The 18th Annual Workshop of the Australasian Language Technology Association. 92--97.
- Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020 a. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020). · arXiv
- 10.18653/v1 · doi:10.18653/v1
- 10.1145/1835449.1835560 · DOI
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
- Pengjie Ren, Zhumin Chen, Zhaochun Ren, Evangelos Kanoulas, Christof Monz, and Maarten de Rijke. 2020. Conversations with search engines. arXiv preprint arXiv:2004.14162 (2020). · arXiv
- 10.1093/jamia/ocaa091 · DOI · Paper page
- Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. Nist Special Publication Sp, Vol. 109 (1995), 109.
- 10.1145/3209978.3210127 · DOI · Paper page
- Yastil Rughbeer, Anban W Pillay, and Edgar Jembere. 2021. Dataset Selection for Transfer Learning in Information Retrieval. In Southern African Conference for Artificial Intelligence Research. Springer, 53--65.
- Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision Vol. 115 3 (2015) 211--252. · DOI · Paper page
- 10.1145/3451964.3451976 · DOI
- Tetsuya Sakai, Sijie Tao, Zhaohao Zeng, Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu, Maria Maistro, Zhicheng Dou, Nicola Ferro, et al. 2020. Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task. Proceedings of NTCIR-15. to appear (2020).
- 10.1145/1076034.1076064 · DOI · Paper page
- Vasileios Stamatis, Leif Azzopardi, and Alan Wilson. 2019. VES Team at TREC Conversational Assistance Track (CAsT) 2019.. In TREC.
- Si Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang, Jie Bao, Zhiyuan Liu, and Paul Bennett. 2020. Meta Adaptive Neural Ranking with Contrastive Synthetic Supervision. arXiv preprint arXiv:2012.14862 (2020). · arXiv
- 10.1145/2983323.2983702 · DOI · Paper page
- Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020. TREC-COVID: constructing a pandemic information retrieval test collection. arXiv preprint arXiv:2005.04474 (2020). · arXiv
- Ellen M Voorhees. 2004. Overview of the TREC 2004 Robust Track. In Trec.
- 10.1145/3397271.3402427 · doi:10.1145/3397271.3402427
- Ellen M Voorhees, Donna K Harman, et al. 2005. TREC: Experiment and evaluation in information retrieval. Vol. 63. MIT press Cambridge, MA.
- Nikos Voskarides Dan Li Andreas Panteli and Pengjie Ren. 2019. ILPS at TREC 2019 Conversational Assistant Track. In TREC. · doi:10.6028/nist.sp.1250.cast-uva.ilps
- 10.1145/3397271.3401130 · DOI
- Disen Wang and Hui Fang. 2019. Exploring Query Reformulation for Conversational Information Seeking.. In TREC. · doi:10.6028/nist.sp.1250.cast-udel_fang
- Lucy Lu Wang and Kyle Lo. 2020. Text mining approaches for dealing with the rapidly expanding literature on COVID-19. Briefings in Bioinformatics (2020).
- Lucy Lu Wang Kyle Lo Yoganand Chandrasekhar Russell Reas Jiangjiang Yang Darrin Eide Kathryn Funk Rodney Kinney Ziyang Liu William Merrill et al. 2020. Cord-19: The covid-19 open research dataset. ArXiv (2020).
- 10.1145/3077136.3080809 · DOI · Paper page
- Chenyan Xiong, Zhenghao Liu, Si Sun, Zhuyun Dai, Kaitao Zhang, Shi Yu, Zhiyuan Liu, Hoifung Poon, Jianfeng Gao, and Paul Bennett. 2020. CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search. arXiv preprint arXiv:2011.01580 (2020). · arXiv
- 10.18653/v1/D19-1352 · DOI · Paper page
- 10.1145/3397271.3401323 · DOI · Paper page
- 10.1145/3159652.3159730 · DOI
- Edwin Zhang, Nikhil Gupta, Rodrigo Nogueira, Kyunghyun Cho, and Jimmy Lin. 2020 a. Rapidly deploying a neural search engine for the covid-19 open research dataset: Preliminary thoughts and lessons learned. arXiv preprint arXiv:2004.05125 (2020). · arXiv
- Edwin Zhang, Nikhil Gupta, Raphael Tang, Xiao Han, Ronak Pradeep, Kuang Lu, Yue Zhang, Rodrigo Nogueira, Kyunghyun Cho, Hui Fang, et al. 2020 b. Covidex: Neural ranking models and keyword search infrastructure for the covid-19 open research dataset. arXiv preprint arXiv:2007.07846 (2020). · arXiv
- 10.1145/3366423.3380131 · DOI
- 10.18653/v1/2020.sustainlp-1.14 · DOI
- Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020. BERT-QE: contextualized query expansion for document re-ranking. arXiv preprint arXiv:2009.07258 (2020). · arXiv
- Mu Zhu. 2004. Recall, Precision and Average Precision. Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, Vol. 2 (2004), 30.
- 10.1145/281250.281256 · DOI · Paper page
Cited by
Large language models in medical and healthcare fields: applications, advances, and challenges
Wang, D., & Zhang, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10921-0 · Cited by 246
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
“A recent study [70] has summarized the promotion effect of MS MARCO on the progress of dense retrieval: “the MS MARCO datasets have enabled large-data exploration of neural models”.”
BackgroundTREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E. M., & Soboroff, I. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463249 · Cited by 58
“Both tasks have large training sets based on human relevance assessments, derived from MS MARCO [3, 10].”
BackgroundInfluentialSelf-citationGEO: Generative Engine Optimization
Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671900 · Cited by 62
“Natural Questions: (Kwiatkowski et al., 2019; Alexander et al., 2022; Craswell et al., 2021) These datasets contain real anonymized user queries from Bing and Google Search Engines.”
BackgroundA Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
Metadata
- Authors
- Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Lin, J.
- Venue
- Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2021-07-11
- Pages
- 1566-1576
- OpenAlex
- W3152624702
- Cited by
- 55
- References
- 11
- FWCI
- 8.53
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: MS MARCO: Benchmarking Ranking Models in the Large-Data Regime. https://app.cochat.ai/papers/10.1145/3404835.3462804
- The paper
- Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Lin, J. (2021). MS MARCO: Benchmarking Ranking Models in the Large-Data Regime. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3404835.3462804
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.