"Deep reinforcement learning for search, recommendation, and online advertising: a survey" by Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin with Martin Vesely as coordinator
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Search, recommendation, and online advertising are the three most important information-providing mechanisms on the web. These information seeking techniques, satisfying users' information needs by suggesting users personalized objects (information or services) at the appropriate time and place, play a crucial role in mitigating the information overload problem. With recent great advances in deep reinforcement learning (DRL), there have been increasing interests in developing DRL based information seeking techniques. These DRL based techniques have two key advantages - (1) they are able to continuously update information seeking strategies according to users' real-time feedback, and (2) they can maximize the expected cumulative long-term reward from users where reward has different definitions according to information seeking applications such as click-through rate, revenue, user satisfaction and engagement. In this paper, we give an overview of deep reinforcement learning for search, recommendation, and online advertising from methodologies to applications, review representative algorithms, and discuss some appealing research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033 · cited 5×
“The contextual bandit model (a.k.a. associative bandits or bandits with side information) is an extension of MAB that takes additional information into account [Auer et al. 2002; Lu et al. 2010].”
BackgroundInfluentialPlanning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · cited 2×
“The Partially Observable Markov Decision Process (POMDP) is an extension of MDP to the case where the state of the system is not necessarily observable [Åström 1965; Smallwood and Sondik 1973; Sondik 1978; Kaelbling et al. 1998].”
Background(2002) · doi:10.1137/s0097539701398375 · Cited by 2,275
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 2×
“The generalization of the Markov Decision Process to the Multi-Agent case is the stochastic game [Bowling and Veloso 2002; Shoham et al. 2003; Busoniu et al. 2008] as: Definition 2.3.”
MethodInfluentialOptimal control of Markov processes with incomplete state information
Åström, K. J. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90154-x · Cited by 854
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 3×
“These methods include popular algorithms such as the Dyna [Sutton 1991], Prioritized Sweeping [Moore and Atkeson 1993], Q -iteration [Buso-niu et al. 2010], Policy Gradient (PG) [Williams 1992], and the variation of PG [Baxter and Bartlett 2001; Kakade 2001].”
MethodPeters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Deep Reinforcement Learning in Large Discrete Action Spaces
Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., & Coppin, B. (2015) · arXiv · doi:10.48550/arxiv.1512.07679 · Cited by 290
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192
Task-Oriented Query Reformulation with Reinforcement Learning
Nogueira, R., & Cho, K. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1061 · Cited by 120 · cited 1×
“In [Nogueira and Cho 2017], RL has been leveraged to solve the query reformulation task: a query reformulation framework is proposed based on a neural network, which rewrites a query to maximize the number of relevant documents returned.”
MethodAn Efficient Bandit Algorithm for Realtime Multivariate Optimization
Hill, D. N., Nassif, H., Liu, Y., Iyer, A., & Vishwanathan, S. V. N. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098184 · Cited by 113 · cited 2×
“In [Hill et al. 2017], bandit formulation is applied to explore the lay-out space efficiently and hill-climbing is used to select optimal content in real-time.”
MethodAutomatic ad format selection via contextual bandits
Tang, L., Rosales, R., Singh, A., & Agarwal, D. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2514700 · Cited by 111 · cited 2×
“…task is typically modeled as multi-armed bandit (MAB) problem with the setting that samples from each arm are iid, feedback is immediate and rewards are stationary [Yang and Lu 2016; Nuara et al. 2018; Gasparini et al. 2018; Tang et al. 2013; Xu et al. 2013; Yuan et al. 2013; Schwartz et al. 2017].”
BackgroundBeyond Ranking: Optimizing Whole-Page Presentation
Wang, Y., Yin, D., Jie, L., Wang, P., Yamada, M., Chang, Y., & Mei, Q. (2016) · Proceedings of the Ninth ACM International Conference on Web Search and Data Mining · doi:10.1145/2835776.2835824 · Cited by 66 · cited 1×
“In [Wang et al. 2016; Wang et al. 2018], a framework is proposed to learn the optimal page presentation to render heterogeneous results onto SERP.”
BackgroundSelf-citation
Cited by
Bias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741
“ys in recommendation have been published recently, focusing on different perspectives of RS. For example, [10] reviews explainable recommendation, [11] reviews knowledgebased recommendation, [12] and [13] summarize the recommendation methods based on deep learning and reinforcement learning, respectively. However, to…”
BackgroundInfluentialReinforcement Learning based Recommender Systems: A Survey
Afsar, M. M., Crump, T., & Far, B. (2023) · ACM Computing Surveys · doi:10.1145/3543846 · Cited by 502
Xu, Z., & Saleh, J. H. (2021) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2021.107530 · Cited by 500
Fan, W., Zhao, X., Li, Q., Derr, T., Ma, Y., Liu, H., Wang, J., & Tang, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3272652 · Cited by 41
Self-citationCHASE: Commonsense-Enriched Advertising on Search Engine with Explicit Knowledge
Zhang, C., Zhou, J., Zang, X., Xu, Q., Yin, L., He, X., Liu, L., Xiong, H., & Dou, D. (2021) · Proceedings of the 30th ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3481902 · Cited by 7
Metadata
- Authors
- Zhao, X., Xia, L., Tang, J., & Yin, D.
- Venue
- ACM SIGWEB Newsletter
- Published
- 2019-07-29
- Volume
- 2019
- Issue
- Spring
- Pages
- 1-15
- ISSN
- 1931-1435 (ISSN-L)
- 1931-1745
- OpenAlex
- W2965512832
- Cited by
- 106
- References
- 15
- FWCI
- 14.37
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.