The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
A modern Bayesian look at the multi‐armed bandit
Abstract
A multi‐armed bandit is an experiment with the goal of accumulating rewards from a payoff distribution with unknown parameters that are to be learned sequentially. This article describes a heuristic for managing multi‐armed bandits called randomized probability matching, which randomly allocates observations to arms according the Bayesian posterior probability that each arm is optimal. Advances in Bayesian computation have made randomized probability matching easy to apply to virtually any payoff distribution. This flexibility frees the experimenter to work with payoff distributions that correspond to certain classical experimental designs that have the potential to outperform methods that are ‘optimal’ in simpler contexts. I summarize the relationships between randomized probability matching and several related heuristics that have been used in the reinforcement learning literature. Copyright © 2010 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Sequential Monte Carlo Methods in Practice
(2001) · Springer eBooks · doi:10.1007/978-1-4757-3437-9 · Cited by 7,366
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
Markov Chains for Exploring Posterior Distributions
Tierney, L. (1994) · The Annals of Statistics · doi:10.1214/aos/1176325750 · Cited by 3,452
Bayesian Analysis of Binary and Polychotomous Response Data
Albert, J. H., & Chib, S. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476321 · Cited by 3,167
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Pfanzagl, J., & Luce, R. D. (1962) · Econometrica · doi:10.2307/1911299 · Cited by 2,469
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
Cited by
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
Machine Learning Methods That Economists Should Know About
Athey, S., & Imbens, G. W. (2019) · Annual Review of Economics · doi:10.1146/annurev-economics-080217-053433 · Cited by 1,163
(2011) · Cited by 1,000
Analysis of Thompson Sampling for the multi-armed bandit problem
Agrawal, S., & Goyal, N. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.1797 · Cited by 819
Online learning: A comprehensive survey
Hoi, S. C. H., Sahoo, D., Lu, J., & Zhao, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.112 · Cited by 746
Thompson Sampling for Contextual Bandits with Linear Payoffs
Agrawal, S., & Goyal, N. (2012) · arXiv · doi:10.48550/arxiv.1209.3352 · Cited by 600
Learning to Optimize via Posterior Sampling
Russo, D., & Van Roy, B. (2014) · Mathematics of Operations Research · doi:10.1287/moor.2014.0650 · Cited by 493
A Tutorial on Thompson Sampling
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z. (2018) · now publishers, Inc. eBooks · doi:10.1561/9781680834710 · Cited by 380
Kaptein, M., Markopoulos, P., de Ruyter, B., & Aarts, E. (2015) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2015.01.004 · Cited by 314
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Kohavi, R., Tang, D., & Xu, Y. (2020) · doi:10.1017/9781108653985 · Cited by 297
Further Optimal Regret Bounds for Thompson Sampling
Agrawal, S., & Goyal, N. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1209.3353 · Cited by 287
Bradford, E., Schweidtmann, A. M., & Lapkin, A. (2018) · Journal of Global Optimization · doi:10.1007/s10898-018-0609-2 · Cited by 266
Customer Acquisition via Display Advertising Using Multi-Armed Bandit Experiments
Schwartz, E. M., Bradlow, E. T., & Fader, P. S. (2017) · Marketing Science · doi:10.1287/mksc.2016.1023 · Cited by 260
Online Display Advertising Markets: A Literature Review and Future Directions
Choi, H., Mela, C. F., Balseiro, S. R., & Leary, A. (2020) · Information Systems Research · doi:10.1287/isre.2019.0902 · Cited by 181
Clarke, H. F., Cardinal, R. N., Rygula, R., Hong, Y. T., Fryer, T. D., Sawiak, S. J., Ferrari, V., Cockcroft, G., Aigbirhio, F. I., Robbins, T. W., et al. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.0718-14.2014 · Cited by 54
“When rewards are drawn from independent Bernoulli random variables (a “binomial bandit”), as in the current situation, the optimality probability (Scott, 2010) is given by:”
BackgroundInfluential
Metadata
- Authors
- Scott, S. L.
- Venue
- Applied Stochastic Models in Business and Industry
- Publisher
- Wiley
- Published
- 2010-11
- Volume
- 26
- Issue
- 6
- Pages
- 639-658
- ISSN
- 1524-1904 (ISSN-L)
- 1526-4025
- OpenAlex
- W2158125716
- Cited by
- 406
- References
- 27
- FWCI
- 13.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.