The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.
Overview of the TREC 2020 deep learning track
Open accessAbstract
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Learning to Match using Local and Distributed Representations of Text for Web Search
Mitra, B., Díaz, F., & Craswell, N. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052579 · Cited by 465
Self-citationUMass at TREC 2004: Novelty and HARD
AbdulJaleel, N., Allan, J., Croft, W. B., Díaz, F., Larkey, L. S., Li, X., Smucker, M. D., & Wade, C. (2004) · doi:10.21236/ada460118 · Cited by 274
From Neural Re-Ranking to Neural Ranking
Zamani, H., Dehghani, M., Croft, W. B., Miller, E. G., & Kamps, J. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271800 · Cited by 212
High accuracy retrieval with multiple nested ranker
Matveeva, I. V., Burges, C. J. C., Burkard, T., Laucius, A., & Wong, L. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148246 · Cited by 133
Optimizing Query Evaluations Using Reinforcement Learning for Web Search
Rosset, C., Jose, D., Ghosh, G., Mitra, B., & Tiwary, S. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210127 · Cited by 31
Self-citation
Research works printed but not linked: 12
- Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 , 2016. · arXiv
- Leonid Boytsov, David Novak, Yury Malkov, and Eric Nyberg. Off the beaten path: Let's replace term-based retrieval with k-nn search. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management , pages 1099-1108. ACM, 2016.
- Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. Orcas: 18 million clicked query-document pairs for analyzing search. arXiv preprint arXiv:2006.05324 , 2020. · arXiv
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 , 2016. · arXiv
- Bhaskar Mitra and Nick Craswell. An updated duet model for passage re-ranking. arXiv preprint arXiv:1903.07666 , 2019. · arXiv
- Bhaskar Mitra, Corby Rosset, David Hawking, Nick Craswell, Fernando Diaz, and Emine Yilmaz. Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks. arXiv preprint arXiv:1907.03693 , 2019. · arXiv
- Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprint arXiv:1904.08375 , 2019. · arXiv
- Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval , 3(4):333-389, 2009.
- Trevor Strohman, Donald Metzler, Howard Turtle, and W Bruce Croft. Indri: A language model-based search engine for complex queries. In Proceedings of the International Conference on Intelligent Analysis , volume 2, pages 2-6. Citeseer, 2005.
- Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval , pages 105-114. ACM, 2011.
- Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. Critically examining the 'neural hype': Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proc. SIGIR , pages 1129-1132. ACM, 2019a.
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 , 2019b. · arXiv
Cited by
Sorted by citations
Items 101–110 of 110
FAA: Fine-grained Attention Alignment for Cascade Document Ranking
Li, Z., Tao, C., Feng, J., Shen, T., Zhao, D., Geng, X., & Jiang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.94
Bruno, W., & Roth, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.369 · Cited by 4
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/728
Learning to Rank: Performance and Practical Barriers to Deployment in Enterprise Search
Daly, C. (2023) · Asia Conference on Information Engineering (ACIE) · doi:10.1109/acie58528.2023.00011
Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval
Yang, J., Liu, Z., Li, C., Sun, G., & Xie, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.223 · Cited by 1
A New Paradigm for Scientific Query AnsweringPreprint
Martinez, J., Ali, W., & Taylor, M. (2024) · Preprints.org · doi:10.20944/preprints202408.2157.v1
Knowledge Graph-Based Question Answering via Enhanced Contrastive LearningPreprint
Smith, M., Ali, W., & Johnson, T. (2024) · Preprints.org · doi:10.20944/preprints202408.2225.v1
Kulkarni, H., MacAvaney, S., Goharian, N., & Frieder, O. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769495
Metadata
- Authors
- Craswell, N., Mitra, B., Yilmaz, E., & Campos, D.
- Affiliations
- Microsoft (United States) (US)
- University of Illinois Urbana-Champaign (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.07662
- OpenAlex
- W3175111331
- Cited by
- 107
- References
- 9
- Citations by year
- 2025: 6 · 2024: 10 · 2023: 38 · 2022: 31 · 2021: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- ad hoc retrieval, document retrieval, passage retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
