The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
Open accessAbstract
Contextual bandit algorithms have become popular for online recommendation systems such as Digg, Yahoo! Buzz, and news recommendation in general. Offline evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "partial-label" nature. Common practice is to create a simulator which simulates the online environment for the problem at hand and then run an algorithm against this simulator. However, creating simulator itself is often difficult and modeling bias is usually unavoidably introduced. In this paper, we introduce a replay methodology for contextual bandit algorithm evaluation. Different from simulator-based approaches, our method is completely data-driven and very easy to adapt to different applications. More importantly, our method can provide provably unbiased evaluations. Our empirical results on a large-scale news article recommendation dataset collected from Yahoo! Front Page conform well with our theoretical results. Furthermore, comparisons between our offline replay and online bucket evaluation of several contextual bandit algorithms show accuracy and effectiveness of our offline evaluation method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,582
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
Bandit Based Monte-Carlo Planning
Kocsis, L., & Szepesvári, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11871842_29 · Cited by 2,855
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Self-citationRCV1: A New Benchmark Collection for Text Categorization Research
Lewis, D., Yang, Y., Rose, T. G., & Li, F. (2004) · Goldsmiths (University of London) · doi:10.5555/1005332.1005345 · Cited by 2,503
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
(2002) · doi:10.1137/s0097539701398375 · Cited by 2,275
Some aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184
Bandit processes and dynamic allocation indices
Gittins, J. C. (1979) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1979.tb01068.x · Cited by 1,487 · cited 1×
“Another class of algorithms are based on Bayes rule, such as Gittins index methods [13].”
MethodReinforcement Learning: An Introduction
(1998) · Kybernetes · doi:10.1108/k.1998.27.9.1093.3 · Cited by 1,472
On the method of bounded differences
McDiarmid, C. (1989) · Surveys in Combinatorics, 1989 · doi:10.1017/cbo9781107359949.008 · Cited by 1,425
Using confidence bounds for exploitation-exploration trade-offs
Auer, P. L. (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944941 · Cited by 1,229
Contextual Bandits with Linear Payoff Functions
Chu, W., Li, L. C., Reyzin, L., & Schapire, R. E. (2011) · Cited by 633
Self-citationApproximately Optimal Approximate Reinforcement Learning
Kakade, S. M., & Langford, J. C. (2002) · Cited by 617
Self-citationBandit Problems, Sequential Allocation of Experiments.
Jones, P. W., Berry, D. A., & Fristedt, B. (1987) · The Journal of the Operational Research Society · doi:10.2307/2582852 · Cited by 566
Sample mean based index policies by O(log n) regret for the multi-armed bandit problem
Agrawal, R. (1995) · Advances in Applied Probability · doi:10.2307/1427934 · Cited by 540
Eligibility Traces for Off-Policy Policy Evaluation
Precup, D., Sutton, R. S., & Singh, S. P. (2000) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 439
PAC model-free reinforcement learning
Strehl, A. L., Li, L., Wiewiora, E., Langford, J., & Littman, M. L. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143955 · Cited by 384
Self-citationThe Epoch-Greedy algorithm for contextual multi-armed bandits
Langford, J. C., & Zhang, T. (2007) · Cited by 320
Self-citationParametric Bandits: The Generalized Linear Case
Filippi, S., Cappé, O., Garivier, A., & Szepesvári, C. (2010) · Cited by 259
Learning from Logged Implicit Exploration DataPreprint
Strehl, A. L., Langford, J. C., Kakade, S. M., & Li, L. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.0120 · Cited by 181
Self-citationContextual Bandit Algorithms with Supervised Learning Guarantees
Beygelzimer, A., Langford, J. C., Li, L. C., Reyzin, L., & Schapire, R. E. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1002.4058 · Cited by 175
Self-citationSome Aspects of the Sequential Design of Experiments
Robbins, H. E. (1985) · Springer eBooks · doi:10.1007/978-1-4612-5110-1_13 · Cited by 168
Approximate Planning in Large POMDPs via Reusable Trajectories
Kearns, M. J., Mansour, Y., & Ng, A. Y. (1999) · neural information processing systems · Cited by 163
Spatio-temporal models for estimating click-through rate
Agarwal, D., Chen, B.-C., & Elango, P. (2009) · International World Wide Web Conference (WWW) · doi:10.1145/1526709.1526713 · Cited by 157
Bandit problems with side observations
Wang, C.-C., Kulkarni, S. R., & Poor, H. V. (2005) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2005.844079 · Cited by 154
Explore/Exploit Schemes for Web Content Optimization
Agarwal, D., Chen, B.-C., & Elango, P. (2009) · Proceedings · doi:10.1109/icdm.2009.52 · Cited by 145
Reinforcement Learning with Immediate Rewards and Linear Hypotheses
Abe, N., Biermann, A., & Long, P. M. (2003) · Algorithmica · doi:10.1007/s00453-003-1038-1 · Cited by 133
Online Models for Content Optimization
Agarwal, D., Chen, B.-C., Elango, P., Motgi, N., Park, S.-T., Ramakrishnan, R., Roy, S., & Zachariah, J. (2008) · neural information processing systems · Cited by 127
RL-Glue: Language-Independent Software for Reinforcement-Learning Experiments
Tanner, B. K., & White, A. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1755857 · Cited by 119
Langford, J. C., Strehl, A. L., & Wortman, J. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390223 · Cited by 112
Self-citationA One-Armed Bandit Problem with a Concomitant Variable
Woodroofe, M. B. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10481033 · Cited by 107
Bandit Problems-Sequential Allocation of Experiments.
Glazebrook, K. D., Berry, D. A., & Fristedt, B. E. (1988) · Journal of the American Statistical Association · doi:10.2307/2288963 · Cited by 53
Associative Reinforcement Learning: Functions in k-DNF
Kaelbling, L. P. (1994) · Machine Learning · doi:10.1023/a:1022689909846 · Cited by 47
Experience-efficient learning in associative bandit problems
Strehl, A. L., Mesterharm, C., Littman, M. L., & Hirsh, H. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143956 · Cited by 37
Online learning for recency search ranking using real-time user feedback
Moon, T., Li, L. C., Chu, W., Liao, C. Y., Zheng, Z., & Chang, Y. (2010) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1871437.1871657 · Cited by 30
Self-citation
Cited by
(2011) · Cited by 1,000
Introduction to Multi-Armed Bandits
Slivkins, A. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000068 · Cited by 780
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531
“Our approach uses inverse propensity scoring (IPS), originally employed in survey sampling [7] and causal inference from observational studies [19], but more recently also in whole page optimization [29], IR evaluation with manual judgments [20], and recommender evaluation [13, 21].”
MethodReinforcement learning improves behaviour from evaluative feedback
Littman, M. L. (2015) · Nature · doi:10.1038/nature14540 · Cited by 368
News recommender systems – Survey and roads ahead
Karimi, M., Jannach, D., & Jugovac, M. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.04.008 · Cited by 336
Metadata
- Authors
- Li, L., Chu, W., Langford, J., & Wang, X.
- Venue
- Proceedings of the fourth ACM international conference on Web search and data mining
- Publisher
- ACM
- Published
- 2011-02-09
- Pages
- 297-306
- OpenAlex
- W2138909795
- Cited by
- 443
- References
- 41
- FWCI
- 46.25
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.