CoChat
Sign Up
Cited by 36Information Retrieval and Search BehaviorRecommender Systems and TechniquesReinforcement Learning in Robotics

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

A Reinforcement Learning Framework for Relevance Feedback

Montazeralghaem, Ali, Zamani, Hamed, Allan, James·2020·Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI 10.1145/3397271.3401099
Open this paper in CoChat — free account

Abstract

We present RML, the first known general reinforcement learning framework for relevance feedback that directly optimizes any desired retrieval metric, including precision-oriented, recall-oriented, and even diversity metrics: RML can be easily extended to directly optimize any arbitrary user satisfaction signal. Using the RML framework, we can select effective feedback terms and weight them appropriately, improving on past methods that fit parameters to feedback algorithms using heuristic approaches or methods that do not directly optimize for retrieval performance. Learning an effective relevance feedback model is not trivial since the true feedback distribution is unknown. Experiments on standard TREC collections compare RML to existing feedback algorithms, demonstrate the effectiveness of RML at optimizing for MAP and α-n DCG, and show the impact on related measures.

Overview

Integrity signals

Registry

Crossref · 10.1145/3397271.3401099

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • To autonomously evolve in the environment, agents have to explore different actions and learn from the feedback.

    “To autonomously evolve in the environment, the agents have to explore different actions and learn from the feedback [92].”

    From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [92]

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Mart237;n Abadi and Agarwal A Barham P TensorFlow . 2016 . Large-scale machine learning on heterogeneous distributed systems . In OSDI'16 . 265--283. Mart237;n Abadi and Agarwal A Barham P TensorFlow. 2016. Large-scale machine learning on heterogeneous distributed systems. In OSDI'16. 265--283.
  2. Nitish Aggarwal and Paul Buitelaar. 2012. Query Expansion Using Wikipedia and Dbpedia.. In CLEF (Online Working Notes/Labs/Workshop). Nitish Aggarwal and Paul Buitelaar. 2012. Query Expansion Using Wikipedia and Dbpedia.. In CLEF (Online Working Notes/Labs/Workshop).
  3. Mohammad Aliannejadi , Hamed Zamani , Fabio Crestani , and W. Bruce Croft . 2018. Target Apps Selection: Towards a Unified Search Framework for Mobile Devices . In SIGIR '18 . 215--224. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2018. Target Apps Selection: Towards a Unified Search Framework for Mobile Devices. In SIGIR '18. 215--224.
  4. Mozhdeh Ariannezhad , Ali Montazeralghaem , Hamed Zamani , and Azadeh Shakery . 2017 . Iterative Estimation of Document Relevance Score for Pseudo-Relevance Feedback. In ECIR'17 . Springer, 676--683. Mozhdeh Ariannezhad, Ali Montazeralghaem, Hamed Zamani, and Azadeh Shakery. 2017. Iterative Estimation of Document Relevance Score for Pseudo-Relevance Feedback. In ECIR'17. Springer, 676--683.
  5. 10.1145/322017.322021 · DOI
  6. Dzmitry Bahdanau , Philemon Brakel , Kelvin Xu , Anirudh Goyal , Ryan Lowe , Joelle Pineau , Aaron Courville , and Yoshua Bengio . 2016. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086 ( 2016 ). Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086 (2016). · arXiv
  7. 10.1145/1390334.1390377 · DOI
  8. Rich Caruana Steve Lawrence and C Lee Giles. 2001. Overfitting in neural nets: Backpropagation conjugate gradient and early stopping. In Advances in neural information processing systems. 402--408. Rich Caruana Steve Lawrence and C Lee Giles. 2001. Overfitting in neural nets: Backpropagation conjugate gradient and early stopping. In Advances in neural information processing systems. 402--408.
  9. 10.1145/1390334.1390446 · DOI
  10. 10.1145/2499178.2499179 · DOI
  11. 10.1145/1645953.1646059 · DOI
  12. 10.1007/s10791-011-9162-z · DOI
  13. 10.1108/eb026683 · DOI
  14. Mostafa Dehghani , Hamed Zamani , Aliaksei Severyn , Jaap Kamps , and W. Bruce Croft . 2017 . Neural Ranking Models with Weak Supervision. In SIGIR'17 . 65--74. Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural Ranking Models with Weak Supervision. In SIGIR'17. 65--74.
  15. 10.1145/2808194.2809491 · DOI
  16. Fernando Diaz , Bhaskar Mitra , and Nick Craswell . 2016. Query expansion with locally-trained word embeddings. arXiv preprint arXiv:1605.07891 ( 2016 ). Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016. Query expansion with locally-trained word embeddings. arXiv preprint arXiv:1605.07891 (2016). · arXiv
  17. Jiafeng Guo , Yixing Fan , Qingyao Ai , and W. Bruce Croft . 2016. A Deep Relevance Matching Model for Ad-hoc Retrieval . In CIKM'16 . 55--64. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-hoc Retrieval. In CIKM'16. 55--64.
  18. 10.1007/978-3-030-15719-7_26 · DOI
  19. 10.1145/775047.775067 · DOI
  20. Diederik P. Kingma and Jimmy Ba . 2015 . Adam : A Method for Stochastic Optimization. In ICLR '15. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR'15.
  21. 10.1145/383952.383970 · DOI
  22. Victor Lavrenko and W. Bruce Croft . 2001 . Relevance Based Language Models. In SIGIR'01 . 120--127. Victor Lavrenko and W. Bruce Croft. 2001. Relevance Based Language Models. In SIGIR'01. 120--127.
  23. Canjia Li , Yingfei Sun , Ben He , Le Wang , Kai Hui , Andrew Yates , Le Sun , and Jungang Xu . 2018 . NPRF: A neural pseudo relevance feedback framework for ad-hoc information retrieval. arXiv preprint arXiv:1810.12936 (2018). Canjia Li, Yingfei Sun, Ben He, Le Wang, Kai Hui, Andrew Yates, Le Sun, and Jungang Xu. 2018. NPRF: A neural pseudo relevance feedback framework for ad-hoc information retrieval. arXiv preprint arXiv:1810.12936 (2018). · arXiv
  24. 10.1145/1645953.1646259 · DOI
  25. 10.1145/1835449.1835546 · DOI
  26. 10.1145/2661829.2661900 · DOI
  27. 10.1145/1277741.1277796 · DOI
  28. 10.1145/2348283.2348356 · DOI
  29. Ali Montazeralghaem , Razieh Rahimi , and James Allan . 2020 . Relevance Ranking Based on Query-Aware Context Analysis. In ECIR'20 . Springer, 446--460. Ali Montazeralghaem, Razieh Rahimi, and James Allan. 2020. Relevance Ranking Based on Query-Aware Context Analysis. In ECIR'20. Springer, 446--460.
  30. 10.1145/2911451.2914768 · DOI
  31. 10.1145/3077136.3080728 · DOI
  32. Ali Montazeralghaem , Hamed Zamani , and Azadeh Shakery . 2018 . Theoretical Analysis of Interdependent Constraints in Pseudo-Relevance Feedback. In SIGIR '18 . ACM, 1249--1252. Ali Montazeralghaem, Hamed Zamani, and Azadeh Shakery. 2018. Theoretical Analysis of Interdependent Constraints in Pseudo-Relevance Feedback. In SIGIR '18. ACM, 1249--1252.
  33. Rodrigo Nogueira and Kyunghyun Cho . 2017. Task-oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572 ( 2017 ). Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572 (2017). · arXiv
  34. Romain Paulus , Caiming Xiong , and Richard Socher . 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 ( 2017 ). Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 (2017). · arXiv
  35. Jay M. Ponte and W. Bruce Croft . 1998. A Language Modeling Approach to Information Retrieval . In SIGIR'98 . 275--281. Jay M. Ponte and W. Bruce Croft. 1998. A Language Modeling Approach to Information Retrieval. In SIGIR'98. 275--281.
  36. Ivaylo Popov , Nicolas Heess , Timothy Lillicrap , Roland Hafner , Gabriel Barth-Maron , Matej Vecerik , Thomas Lampe , Yuval Tassa , Tom Erez , and Martin Riedmiller . 2017. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03073 ( 2017 ). Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. 2017. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03073 (2017). · arXiv
  37. Marc'Aurelio Ranzato , Sumit Chopra , Michael Auli , and Wojciech Zaremba . 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 ( 2015 ). Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 (2015). · arXiv
  38. 10.1109/CVPR.2017.131 · DOI
  39. Stephen E. Robertson and Karen Sparck Jones . 1988 . Document Retrieval Systems. Taylor Graham Publishing , London, UK, UK, Chapter Relevance Weighting of Search Terms, 143--160. http://dl.acm.org/citation.cfm?id=106765.106783 Stephen E. Robertson and Karen Sparck Jones. 1988. Document Retrieval Systems. Taylor Graham Publishing, London, UK, UK, Chapter Relevance Weighting of Search Terms, 143--160. http://dl.acm.org/citation.cfm?id=106765.106783
  40. J. J. Rocchio . 1971. Relevance Feedback in Information Retrieval. The SMART Retrieval System: Experiments in Automatic Document Processing . Prentice Hall , 313--323. J. J. Rocchio. 1971. Relevance Feedback in Information Retrieval. The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice Hall, 313--323.
  41. Corby Rosset , Damien Jose , Gargi Ghosh , Bhaskar Mitra , and Saurabh Tiwary . 2018. Optimizing Query Evaluations using Reinforcement Learning for Web Search. arXiv preprint arXiv:1804.04410 ( 2018 ). Corby Rosset, Damien Jose, Gargi Ghosh, Bhaskar Mitra, and Saurabh Tiwary. 2018. Optimizing Query Evaluations using Reinforcement Learning for Web Search. arXiv preprint arXiv:1804.04410 (2018). · arXiv
  42. 10.1002/(SICI)1097-4571(199006)41:4<288::AID-ASI8>3.0.CO;2-H · DOI
  43. 10.1145/2766462.2767738 · DOI
  44. David Silver , Julian Schrittwieser , Karen Simonyan , Ioannis Antonoglou , Aja Huang , Arthur Guez , Thomas Hubert , Lucas Baker , Matthew Lai , Adrian Bolton , and others. 2017 . Mastering the game of Go without human knowledge . Nature , Vol. 550 , 7676 (2017), 354. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, and others. 2017. Mastering the game of Go without human knowledge. Nature, Vol. 550, 7676 (2017), 354.
  45. Richard S Sutton and Andrew G Barto . 2018. Reinforcement learning: An introduction . MIT press . Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
  46. Yuan Tang . 2016. TF. Learn: TensorFlow's high-level module for distributed machine learning. arXiv preprint arXiv:1612.04251 ( 2016 ). Yuan Tang. 2016. TF. Learn: TensorFlow's high-level module for distributed machine learning. arXiv preprint arXiv:1612.04251 (2016). · arXiv
  47. 10.1145/3077136.3080685 · DOI
  48. Ronald J Williams . 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , Vol. 8 , 3--4 ( 1992 ), 229--256. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, Vol. 8, 3--4 (1992), 229--256.
  49. Liu Yang , Minghui Qiu , Chen Qu , Jiafeng Guo , Yongfeng Zhang , W Bruce Croft , Jun Huang , and Haiqing Chen . 2018. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. arXiv preprint arXiv:1805.00188 ( 2018 ). Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W Bruce Croft, Jun Huang, and Haiqing Chen. 2018. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. arXiv preprint arXiv:1805.00188 (2018). · arXiv
  50. 10.1145/2970398.2970405 · DOI
  51. Hamed Zamani and W. Bruce Croft . 2017 . Relevance-based Word Embedding. In SIGIR'17 . 505--514. Hamed Zamani and W. Bruce Croft. 2017. Relevance-based Word Embedding. In SIGIR'17. 505--514.
  52. Hamed Zamani , Javid Dadashkarimi , Azadeh Shakery , and W. Bruce Croft . 2016 . Pseudo-Relevance Feedback Based on Matrix Factorization. In CIKM'16 . Hamed Zamani, Javid Dadashkarimi, Azadeh Shakery, and W. Bruce Croft. 2016. Pseudo-Relevance Feedback Based on Matrix Factorization. In CIKM'16.
  53. Wei Zeng , Jun Xu , Yanyan Lan , Jiafeng Guo , and Xueqi Cheng . 2018 . Multi Page Search with Reinforcement Learning to Rank . In SIGIR '18 . ACM, 175--178. Wei Zeng, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2018. Multi Page Search with Reinforcement Learning to Rank. In SIGIR '18. ACM, 175--178.
  54. 10.1145/502585.502654 · DOI
  55. 10.1145/984321.984322 · DOI

Cited by

Sorted by citations

Metadata

Authors
Montazeralghaem, A., Zamani, H., & Allan, J.
Affiliations
  • University of Massachusetts Amherst (US)
  • Microsoft (United States) (US)
ORCID
Venue
Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2020-07-25
Pages
59-68
DOI
10.1145/3397271.3401099
OpenAlex
W3034329167
Cited by
36
References
2
Citations by year
2025: 4 · 2024: 7 · 2023: 8 · 2022: 7 · 2021: 8 · 2020: 2
FWCI
4.67
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
Funders
  • Center for Intelligent Information Retrieval (CIIR) University of Massachusetts Amherst
  • National Science Foundation (NSF) (IIS-1617408)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
relevance feedback, reinforcement learning, retrieval evaluation metrics, diversity indices, maps, TREC collections

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review