Overview of the TREC 2019 deep learning track
Open accessAbstract
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries. This year 15 groups submitted a total of 75 runs, using various combinations of deep learning, transfer learning and traditional IR ranking methods. Deep learning runs significantly outperformed traditional IR runs. Possible explanations for this result are that we introduced large training data and we included deep models trained on such data in our judging pools, whereas some past studies did not have such training data or pooling.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM assessment is an effective alternative to human-based assessment.
“Our experiments with the TREC Deep Learning (DL) Tracks from 2019-2023 [4-8] verified the claims by Thomas et al. [21] about LLM assessment being an effective alternative to humanbased assessment, using OpenAI's latest model GPT-4o.”
From UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545 · see citing works
Flagged first, then by citations
Research works printed but not linked: 28
- Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. A system for efficient high-recall retrieval. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 1317-1320, 2018.
- Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 , 2016. · arXiv
- Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.
- Leonid Boytsov, David Novak, Yury Malkov, and Eric Nyberg. Off the beaten path: Let's replace term-based retrieval with k-nn search. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management , pages 1099-1108. ACM, 2016.
- Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees. Bias and the limits of pooling for large collections. Information retrieval , 10(6):491-508, 2007.
- Gordon V Cormack and Maura R Grossman. Beyond pooling. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 1169-1172, 2018.
- Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft. Neural ranking models with weak supervision. In Proc. SIGIR , pages 65-74. ACM, 2017.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. CVPR , pages 248-255. Ieee, 2009.
- Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. Trec complex answer retrieval overview. In Proceedings of TREC , 2017.
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.
- Tie-Yan Liu. Learning to rank for information retrieval. Foundation and Trends in Information Retrieval , 3(3):225331, March 2009.
- Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9 (Nov):2579-2605, 2008.
- Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong. High accuracy retrieval with multiple nested ranker. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval , pages 437-444. ACM, 2006.
- Bhaskar Mitra and Nick Craswell. An updated duet model for passage re-ranking. arXiv preprint arXiv:1903.07666 , 2019. · arXiv
- Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In Proc. WWW , pages 1291-1299, 2017.
- Bhaskar Mitra, Corby Rosset, David Hawking, Nick Craswell, Fernando Diaz, and Emine Yilmaz. Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks. arXiv preprint arXiv:1907.03693 , 2019. · arXiv
- Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprint arXiv:1904.08375 , 2019. · arXiv
- Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends R © in Information Retrieval , 3(4):333-389, 2009.
- Corby Rosset, Damien Jose, Gargi Ghosh, Bhaskar Mitra, and Saurabh Tiwary. Optimizing query evaluations using reinforcement learning for web search. In Proc. SIGIR , pages 1193-1196. ACM, 2018.
- Trevor Strohman, Donald Metzler, Howard Turtle, and W Bruce Croft. Indri: A language model-based search engine for complex queries. In Proceedings of the International Conference on Intelligent Analysis , volume 2, pages 2-6. Citeseer, 2005.
- Ellen M Voorhees. On building fair and reusable test collections using bandit techniques. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management , pages 407-416, 2018.
- Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval , pages 105-114. ACM, 2011.
- Peilin Yang and Jimmy Lin. Reproducing and generalizing semantic term matching in axiomatic information retrieval. In European Conference on Information Retrieval , pages 369-381. Springer, 2019.
- Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. Critically examining the 'neural hype': Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proc. SIGIR , pages 1129-1132. ACM, 2019a.
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 , 2019b. · arXiv
- Hamed Zamani, Mostafa Dehghani, W Bruce Croft, Erik Learned-Miller, and Jaap Kamps. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In Proc. CIKM , pages 497-506. ACM, 2018.
- Zhaohao Zeng and Tetsuya Sakai. Bm25 pseudo relevance feedback using anserini at waseda university. In Proceedings The Open-Source IR Replicability Challenge (OSIRRC) Workshop , 2019.
- Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval , pages 307-314, 1998.
Cited by
Sorted by citations
Sparse, Dense, and Attentional Representations for Text Retrieval
Luan, Y., Eisenstein, J., Toutanova, K., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00369 · Cited by 363 · see citing works
Optimizing Dense Retrieval Model Training with Hard Negatives
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462880 · Cited by 248 · see citing works
Generation-Augmented Retrieval for Open-Domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.316 · Cited by 146 · see citing works
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28 · see citing works
Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
Tang, H., Sun, X., Jin, B., Wang, J., Zhang, F., & Wu, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.392 · Cited by 26 · see citing works
Recent Advances in Generative Information Retrieval
Tang, Y., Zhang, R., Guo, J., & de Rijke, M. (2023) · Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region · doi:10.1145/3624918.3629547 · Cited by 22 · see citing works
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16 · see citing works
“These judgments were originally provided by human NIST assessors who were given the query and the following relevance scale [7]: 0 Irrelevant: The passage has nothing to do with the query 1 Related: The passage seems related to the query but does not answer it 2 Highly relevant: The passage has…”
MethodSelf-citationOptimizing Dense Retrieval Model Training with Hard Negatives
Zhan, J., Mao, J., Liu, Y., Guo, J., Zhang, M., & Ma, S. (2021) · arXiv · doi:10.48550/arxiv.2104.08051 · Cited by 251 · see citing works
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · see citing works
“We use the datasets with annotations from the TREC Passage Re-ranking challenge (Craswell et al., 2020).”
MethodSame Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks
Panthi, S., & Abdelfattah, R. (2026) · arXiv · doi:10.48550/arxiv.2605.24060
“The closest analogy is classical IR work on evaluation sensitivity (Bailey et al., 2008; Yang et al., 2019; Craswell et al., 2020; Voorhees, 2001; Voorhees and Harman, 2005; Manning et al., 2008; Järvelin and Käkäläinen, 2002).”
- Authors
- Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Voorhees, E. M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.07820
- OpenAlex
- W3130740619
- Cited by
- 68
- References
- 2
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 5 · 2022: 11 · 2021: 34 · 2020: 10 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- ad hoc retrieval, document retrieval, passage retrieval, large-scale training data, blind evaluation, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
