The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Neural Ranking Models with Weak Supervision
Open accessAbstract
Despite the impressive improvements achieved by unsupervised deep neural networks in computer vision and NLP tasks, such improvements have not yet been observed in ranking for information retrieval. The reason may be the complexity of the ranking problem, as it is not obvious how to learn from queries and documents when no supervised signal is available. Hence, in this paper, we propose to train a neural ranking model using weak supervision, where labels are obtained automatically without human annotators or any external resources (e.g., click data). To this aim, we use the output of an unsupervised ranking model, such as BM25, as a weak supervision signal. We further train a set of simple yet effective ranking models based on feed-forward neural networks. We study their effectiveness under various learning scenarios (point-wise and pair-wise models) and using different input representations (i.e., from encoding query-document pairs into dense/sparse vectors to using word embedding representation). We train our networks using tens of millions of training instances and evaluate it on two standard collections: a homogeneous news collection (Robust) and a heterogeneous large-scale web collection (ClueWeb). Our experiments indicate that employing proper objective functions and letting the networks to learn the input representation based on weakly supervised data leads to impressive performance, with over 13% and 35% MAP improvements over the BM25 model on the Robust and the ClueWeb collections. Our findings also suggest that supervised neural ranking models can greatly benefit from pre-training on large amounts of weakly labeled data that can be easily obtained from unsupervised IR models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Optimizing search engines using clickthrough data
Joachims, T. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775067 · Cited by 3,873
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Samy Bengio, A. (2010) · Journal of Machine Learning Research · Cited by 2,631
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · International Conference on Artificial Intelligence and Statistics · doi:10.5555/1756006.1756025 · Cited by 2,579
SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1994) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812797926_0003 · Cited by 2,350
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009
(2011) · Cited by 1,258
Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations
Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., & Weld, D. S. (2011) · Cited by 874
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983769 · Cited by 806
Self-citationLearning semantic representations using convolutional neural networks for web search
Shen, Y., He, X., Gao, J., Deng, L., & Mesnil, G. (2014) · Proceedings of the 23rd International Conference on World Wide Web · doi:10.1145/2567948.2577348 · Cited by 670
Pass, G., Chowdhury, A., & Torgeson, C. (2006) · Proceedings of the 1st international conference on Scalable information systems - InfoScale '06 · doi:10.1145/1146847.1146848 · Cited by 666
Twitter Sentiment Analysis with Deep Convolutional Neural Networks
Severyn, A., & Moschitti, A. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767830 · Cited by 659
Self-citationSupport vector learning for ordinal regression
Herbrich, R. (1999) · 9th International Conference on Artificial Neural Networks: ICANN '99 · doi:10.1049/cp:19991091 · Cited by 448
Learning to Match using Local and Distributed Representations of Text for Web Search
Mitra, B., Diaz, F., & Craswell, N. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052579 · Cited by 426
Liu, X., Gao, J., He, X., Deng, L., Duh, K., & Wang, Y.-Y. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1092 · Cited by 395
Efficient and effective spam filtering and re-ranking for large web datasets
Cormack, G. V., Smucker, M. D., & Clarke, C. L. A. (2011) · Information Retrieval · doi:10.1007/s10791-011-9162-z · Cited by 282
A Deep Architecture for Matching Short Texts
Lu, Z., & Li, H. (2013) · Cited by 225
A Neural Click Model for Web Search
Borisov, A., Markov, I., de Rijke, M., & Serdyukov, P. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2883033 · Cited by 187
aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model
Yang, L., Ai, Q., Guo, J., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983818 · Cited by 176
Self-citationLearning Latent Vector Spaces for Product Search
Gysel, C. V., de Rijke, M., & Kanoulas, E. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983702 · Cited by 149
Embedding-based Query Language Models
Zamani, H., & Croft, W. B. (2016) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/2970398.2970405 · Cited by 148
Self-citationEfficient Ranking from Pairwise Comparisons
Wauthier, F. L., Jordan, M. I., & Jojić, N. (2013) · Cited by 141
Learning to Reweight Terms with Distributed Representations
Zheng, G., & Callan, J. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767700 · Cited by 120
Unsupervised, Efficient and Semantic Expertise Retrieval
Gysel, C. V., de Rijke, M., & Worring, M. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2882974 · Cited by 78
Situational Context for Ranking in Personal Search
Zamani, H., Bendersky, M., Wang, X., & Zhang, M. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052648 · Cited by 66
Self-citationEnd to End Long Short Term Memory Networks for Non-Factoid Question Answering
Cohen, D., & Croft, W. B. (2016) · doi:10.1145/2970398.2970438 · Cited by 51
Self-citationPseudo test collections for learning web search ranking functions
Asadi, N., Metzler, D., Elsayed, T., & Lin, J. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010058 · Cited by 40
Global Distant Supervision for Relation Extraction
Han, X., & Sun, L. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10377 · Cited by 37
Improving Distant Supervision for Information Extraction Using Label Propagation Through Lists
Bing, L., Chaudhari, S., Wang, R., et al. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1060 · Cited by 30
Generalizing Translation Models in the Probabilistic Relevance Framework
Rekabsaz, N., Lupu, M., Hanbury, A., & Zuccon, G. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983833 · Cited by 23
Relevance-based Word EmbeddingPreprint
Zamani, H., & Croft, W. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.03556 · Cited by 22
Self-citationGetting Started with Neural Models for Semantic Matching in Web SearchPreprint
Onal, K. D., Altıngövde, İ. S., Karagöz, P., & de Rijke, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03305 · Cited by 7
Cited by
A clinical text classification paradigm using weak supervision and deep representation
Wang, Y., Sohn, S., Liu, S., Shen, F., Wang, L., Atkinson, E. J., Amin, S., & Liu, H. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0723-6 · Cited by 405
Deeper Text Understanding for IR with Contextual Neural Language Modeling
Dai, Z., & Callan, J. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331303 · Cited by 402
Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search
Dai, Z., Xiong, C., Callan, J., & Liu, Z. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159659 · Cited by 341
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
Nie, Y., Wang, S., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1258 · Cited by 132
Distilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, E. (2020) · arXiv · doi:10.48550/arxiv.2012.04584 · Cited by 102
“We consider two other objectives which have been used in Dehghani et al. (2017), where BM25 is used as a teacher model to train a neural ranker.”
Method
Metadata
- Authors
- Dehghani, M., Zamani, H., Severyn, A., Kamps, J., & Croft, W. B.
- Venue
- Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2017-08-07
- Pages
- 65-74
- OpenAlex
- W2610935556
- Cited by
- 299
- References
- 39
- FWCI
- 35.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.