The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
A Tutorial on Thompson Sampling
Abstract
Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance. The algorithm addresses a broad range of problems in a computationally efficient manner and is therefore enjoying wide use. A Tutorial on Thompson Sampling covers the algorithm and its application, illustrating concepts through a range of examples, including Bernoulli bandit problems, shortest path problems, product recommendation, assortment, active learning with neural networks, and reinforcement learning in Markov decision processes. Most of these problems involve complex information structures, where information revealed by taking an action informs beliefs about other actions. It also discusses when and why Thompson sampling is or is not effective and relations to alternative algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
Casella, G., & George, E. I. (1992) · The American Statistician · doi:10.1080/00031305.1992.10475878 · Cited by 2,364
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. W. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2182033 · Cited by 1,850
Bayesian Learning via Stochastic Gradient Langevin Dynamics
Welling, M., & Teh, Y. W. (2012) · Cited by 1,551
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems
Bubeck, S. (2012) · now publishers, Inc. eBooks · doi:10.1561/9781601986276 · Cited by 1,404
(2011) · Cited by 1,000
Exponential Convergence of Langevin Distributions and Their Discrete Approximations
Roberts, G. O., & Tweedie, R. L. (1996) · Bernoulli · doi:10.2307/3318418 · Cited by 998
Multi-Armed Bandit Allocation Indices.
Bather, J. A., & Gittins, J. C. (1990) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982809 · Cited by 998
An experimental comparison of click position-bias models
Craswell, N., Zoeter, O., Taylor, M., & Ramsey, B. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341545 · Cited by 942
Improved Algorithms for Linear Stochastic Bandits
Abbasi-Yadkori, Y., Pál, D., & Szepesvári, C. (2011) · Cited by 919
Casella, G., & George, E. I. (1992) · The American Statistician · doi:10.2307/2685208 · Cited by 863
Analysis of Thompson Sampling for the multi-armed bandit problem
Agrawal, S., & Goyal, N. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.1797 · Cited by 819
Near-optimal Regret Bounds for Reinforcement Learning
Jaksch, T., Ortner, R., & Auer, P. L. (2010) · Cited by 784
Multi‐Armed Bandit Allocation Indices
Gittins, J., Glazebrook, K., & Weber, R. (2011) · Wiley eBooks · doi:10.1002/9780470980033 · Cited by 684
Optimal Scaling of Discrete Approximations to Langevin Diffusions
Roberts, G. O., & Rosenthal, J. S. (1998) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00123 · Cited by 632
Thompson Sampling for Contextual Bandits with Linear Payoffs
Agrawal, S., & Goyal, N. (2012) · arXiv · doi:10.48550/arxiv.1209.3352 · Cited by 600
(2008) · ScholarlyCommons (University of Pennsylvania) · Cited by 581
Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise
Mattingly, J. C., Stuart, A. M., & Higham, D. J. (2002) · Stochastic Processes and their Applications · doi:10.1016/s0304-4149(02)00150-3 · Cited by 550
Learning to Optimize via Posterior Sampling
Russo, D., & Van Roy, B. (2014) · Mathematics of Operations Research · doi:10.1287/moor.2014.0650 · Cited by 493
Self-citationA Knowledge-Gradient Policy for Sequential Information Collection
Frazier, P. I., Powell, W. B., & Dayanik, S. (2008) · SIAM Journal on Control and Optimization · doi:10.1137/070693424 · Cited by 447
(2010) · Cited by 444
Deep Exploration via Bootstrapped DQN
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 434
Self-citationLinearly Parameterized Bandits
Rusmevichientong, P., & Tsitsiklis, J. N. (2010) · Mathematics of Operations Research · doi:10.1287/moor.1100.0446 · Cited by 427
The Knowledge-Gradient Policy for Correlated Normal Beliefs
Frazier, P., Powell, W., & Dayanik, S. (2009) · INFORMS Journal on Computing · doi:10.1287/ijoc.1080.0314 · Cited by 408
A modern Bayesian look at the multi‐armed bandit
Scott, S. L. (2010) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.874 · Cited by 406
Multi-armed bandits in metric spaces
Kleinberg, R., Slivkins, A., & Upfal, E. (2008) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1374376.1374475 · Cited by 396
Reinforcement learning improves behaviour from evaluative feedback
Littman, M. L. (2015) · Nature · doi:10.1038/nature14540 · Cited by 368
Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis
Kaufmann, E., Korda, N., & Munos, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-34106-9_18 · Cited by 365
A Bayesian Framework for Reinforcement Learning
Strens, M. J. A. (2000) · International Conference on Machine Learning · Cited by 358
Deep Exploration via Randomized Value FunctionsPreprint
Osband, I., Russo, D., Wen, Z., & Roy, B. V. (2017) · Journal of machine learning research · doi:10.48550/arxiv.1703.07608 · Cited by 350
Self-citationThe Multi-Armed Bandit Problem: Decomposition and Computation
Katehakis, M. N., & Veinott, A. F. (1987) · Mathematics of Operations Research · doi:10.1287/moor.12.2.262 · Cited by 333
Further Optimal Regret Bounds for Thompson Sampling
Agrawal, S., & Goyal, N. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1209.3353 · Cited by 287
Customer Acquisition via Display Advertising Using Multi-Armed Bandit Experiments
Schwartz, E. M., Bradlow, E. T., & Fader, P. S. (2017) · Marketing Science · doi:10.1287/mksc.2016.1023 · Cited by 260
Cascading Bandits: Learning to Rank in the Cascade ModelPreprint
Kveton, B., Szepesvári, C., Wen, Z., & Ashkan, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.02763 · Cited by 247
Self-citationOnline Network Revenue Management Using Thompson Sampling
Ferreira, K. J., Simchi-Levi, D., & Wang, H. (2018) · Operations Research · doi:10.1287/opre.2018.1755 · Cited by 243
Bayesian Reinforcement Learning: A Survey
Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2015) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000049 · Cited by 234
(More) Efficient Reinforcement Learning via Posterior Sampling
Osband, I., Russo, D., & Roy, B. V. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0940 · Cited by 222
Self-citationBubeck, S., Munos, R., Stoltz, G., & Szepesvári, C. (2011) · Journal of Machine Learning Research · Cited by 214
On Bayesian Upper Confidence Bounds for Bandit Problems
Kaufmann, E., Garivier, A., & Paristech, T. (2012) · Cited by 213
A dynamic allocation index for the discounted multiarmed bandit problem
GITTINS, J. C., & JONES, D. M. (1979) · Biometrika · doi:10.1093/biomet/66.3.561 · Cited by 207
Near-optimal Regret Bounds for Reinforcement Learning
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859902 · Cited by 207
Generalization and Exploration via Randomized Value FunctionsPreprint
Osband, I., Roy, B. V., & Wen, Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.0635 · Cited by 194
Self-citationLinear Thompson sampling revisited
Abeille, M., & Lazaric, A. (2017) · Electronic Journal of Statistics · doi:10.1214/17-ejs1341si · Cited by 179
Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards
Besbes, O., Gur, Y., & Zeevi, A. J. (2014) · doi:10.13140/rg.2.1.2862.3844 · Cited by 168
Parallelised Bayesian Optimisation via Thompson Sampling
Kandasamy, K., Krishnamurthy, A., Schneider, J., & Póczos, B. (2018) · International Conference on Artificial Intelligence and Statistics · Cited by 154
On the Theory of Apportionment
Thompson, W. R. (1935) · American Journal of Mathematics · doi:10.2307/2371219 · Cited by 142
Learning to Optimize via Information-Directed Sampling
Russo, D., & Roy, B. V. (2017) · Operations Research · doi:10.1287/opre.2017.1663 · Cited by 141
Self-citationWhy is Posterior Sampling Better than Optimism for Reinforcement Learning?Preprint
Osband, I., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.00215 · Cited by 141
Self-citationSimple Bayesian Algorithms for Best-Arm Identification
Russo, D. (2020) · Operations Research · doi:10.1287/opre.2019.1911 · Cited by 140
Self-citationThompson Sampling for Complex Online Problems
Gopalan, A., Mannor, S., & Mansour, Y. (2014) · Cited by 135
Efficient Thompson sampling for online matrix-factorization recommendation
Kawale, J., Bui, H., Kveton, B., Thanh, L. T., & Chawla, S. (2015) · ePrints Soton (University of Southampton) · doi:10.5555/2969239.2969384 · Cited by 115
An Efficient Bandit Algorithm for Realtime Multivariate Optimization
Hill, D. N., Nassif, H., Liu, Y., Iyer, A., & Vishwanathan, S. V. N. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098184 · Cited by 113
Consistency and fluctuations for stochastic gradient Langevin dynamics
(2016) · Warwick Research Archive Portal (University of Warwick) · doi:10.5555/2946645.2946652 · Cited by 104
Model-based Reinforcement Learning and the Eluder DimensionPreprint
Osband, I., & Roy, B. V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.1853 · Cited by 103
Self-citationLASER: a scalable response prediction platform for online advertising
Agarwal, D., Long, B., Traupman, J., Xin, D., & Zhang, L. (2014) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2556195.2556252 · Cited by 100
Sampling from a Log-Concave Distribution with Projected Langevin Monte Carlo
Bubeck, S., Eldan, R., & Lehec, J. (2018) · Discrete & Computational Geometry · doi:10.1007/s00454-018-9992-1 · Cited by 96
Multi‐armed bandit experiments in the online service economy
Scott, S. L. (2015) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.2104 · Cited by 94
Thompson Sampling for Stochastic Control: The Finite Parameter Case
Kim, M. J. (2017) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2017.2653942 · Cited by 82
Exploration and Inference in Learning from Reinforcement
Wyatt, J. (1998) · ERA · Cited by 75
Eluder Dimension and the Sample Complexity of Optimistic Exploration
Russo, D., & Roy, B. V. (2013) · neural information processing systems · Cited by 75
Self-citationLearning Unknown Markov Decision Processes: A Thompson Sampling ApproachPreprint
Ouyang, Y., Gagrani, M., Nayyar, A., & Jain, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04570 · Cited by 73
Thompson Sampling for Learning Parameterized Markov Decision ProcessesPreprint
Gopalan, A., & Mannor, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.7498 · Cited by 69
Near-optimal Reinforcement Learning in Factored MDPsPreprint
Osband, I., & Roy, B. V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.3741 · Cited by 63
Self-citationThompson Sampling for the MNL-BanditPreprint
Agrawal, S., Avadhanula, V., Goyal, V., & Zeevi, A. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00977 · Cited by 51
Optimality of Thompson Sampling for Gaussian Bandits Depends on PriorsPreprint
Honda, J., & Akimichi Takemura, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.1894 · Cited by 46
Lu, X., & Van Roy, B. (2017) · arXiv · doi:10.48550/arxiv.1705.07347 · Cited by 42
Satisficing in Time-Sensitive Bandit Learning
Russo, D., & Roy, B. V. (2022) · Mathematics of Operations Research · doi:10.1287/moor.2021.1229 · Cited by 42
Self-citationInformation Directed Sampling for Stochastic Bandits With Graph Feedback
Liu, F., Buccapatnam, S., & Shroff, N. B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11751 · Cited by 36
Thompson Sampling: An Asymptotically Optimal Finite Time AnalysisPreprint
Kaufmann, E., Korda, N., & Munos, R. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.4217 · Cited by 33
Thompson sampling with the online bootstrapPreprint
Eckles, D., & Kaptein, M. C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4009 · Cited by 28
Bayesian mixture modelling and inference based Thompson sampling in Monte-Carlo tree search
Bai, A., Wu, F., & Chen, X. (2013) · ePrints Soton (University of Southampton) · Cited by 22
Exploiting the Natural Exploration In Contextual Bandits.Preprint
Bastani, H., Bayati, M., & Khosravi, K. (2017) · arXiv (Cornell University) · Cited by 10
Cited by
Introduction to Multi-Armed Bandits
Slivkins, A. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000068 · Cited by 780
Emerging materials intelligence ecosystems propelled by machine learning
Batra, R., Song, L., & Ramprasad, R. (2020) · Nature Reviews Materials · doi:10.1038/s41578-020-00255-y · Cited by 446
On the Sample Complexity of the Linear Quadratic Regulator
Dean, S., Mania, H., Matni, N., Recht, B., & Tu, S. (2020) · Foundations of Computational Mathematics · doi:10.1007/s10208-019-09426-y · Cited by 436
(2019) · International Conference on Machine Learning · Cited by 49
Reward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35
Metadata
- Authors
- Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z.
- Venue
- now publishers, Inc. eBooks
- Publisher
- now Publishers Inc
- Published
- 2018
- OpenAlex
- W2963007936
- Cited by
- 380
- References
- 77
- FWCI
- 44.99
- Type
- book
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.