The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
A modern Bayesian look at the multi‐armed bandit
Abstract
A multi‐armed bandit is an experiment with the goal of accumulating rewards from a payoff distribution with unknown parameters that are to be learned sequentially. This article describes a heuristic for managing multi‐armed bandits called randomized probability matching, which randomly allocates observations to arms according the Bayesian posterior probability that each arm is optimal. Advances in Bayesian computation have made randomized probability matching easy to apply to virtually any payoff distribution. This flexibility frees the experimenter to work with payoff distributions that correspond to certain classical experimental designs that have the potential to outperform methods that are ‘optimal’ in simpler contexts. I summarize the relationships between randomized probability matching and several related heuristics that have been used in the reinforcement learning literature. Copyright © 2010 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Sequential Monte Carlo Methods in Practice
(2001) · Springer eBooks · doi:10.1007/978-1-4757-3437-9 · Cited by 7,366
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033
Markov Chains for Exploring Posterior Distributions
Tierney, L. (1994) · The Annals of Statistics · doi:10.1214/aos/1176325750 · Cited by 3,452
Bayesian Analysis of Binary and Polychotomous Response Data
Albert, J. H., & Chib, S. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476321 · Cited by 3,167
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Pfanzagl, J., & Luce, R. D. (1962) · Econometrica · doi:10.2307/1911299 · Cited by 2,469
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
Bayesian Forecasting and Dynamic Models
(1997) · Springer Series in Statistics · doi:10.1007/b98971 · Cited by 1,917
Bayesian Experimental Design: A Review
Chaloner, K., & Verdinelli, I. (1995) · Statistical Science · doi:10.1214/ss/1177009939 · Cited by 1,839
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2007) · Wiley Series in Probability and Statistics · doi:10.1002/9780470182963 · Cited by 1,566
Bandit processes and dynamic allocation indices
Gittins, J. C. (1979) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1979.tb01068.x · Cited by 1,487
Bayesian Forecasting and Dynamic Models
West, M., & Harrison, J. (1989) · Springer Series in Statistics · doi:10.1007/978-1-4757-9365-9 · Cited by 1,295
Restless bandits: activity allocation in a changing world
Whittle, P. (1988) · Journal of Applied Probability · doi:10.1017/s0021900200040420 · Cited by 1,173
Berry, D. A., & Fristedt, B. (1985) · Springer eBooks · doi:10.1007/978-94-015-3711-7 · Cited by 633
Bandit Problems, Sequential Allocation of Experiments.
Jones, P. W., Berry, D. A., & Fristedt, B. (1987) · The Journal of the Operational Research Society · doi:10.2307/2582852 · Cited by 566
The Theory of the Design of Experiments
Cox, D. R., & Reid, N. (2000) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420035834 · Cited by 559
Sample mean based index policies by O(log n) regret for the multi-armed bandit problem
Agrawal, R. (1995) · Advances in Applied Probability · doi:10.2307/1427934 · Cited by 540
Adaptive Treatment Allocation and the Multi-Armed Bandit Problem
Lai, T. L. (1987) · The Annals of Statistics · doi:10.1214/aos/1176350495 · Cited by 279
A PROBLEM IN THE SEQUENTIAL DESIGN OF EXPERIMENTS
Bellman, R. (1954) · Cited by 190
On the Theory of Apportionment
Thompson, W. R. (1935) · American Journal of Mathematics · doi:10.2307/2371219 · Cited by 142
Optimal learning and experimentation in bandit problems
Brezzi, M., & Lai, T. L. (2002) · Journal of Economic Dynamics and Control · doi:10.1016/s0165-1889(01)00028-8 · Cited by 109
A SEQUENTIAL DECISION PROBLEM WITH A FINITE MEMORY
Robbins, H. E. (1956) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.42.12.920 · Cited by 86
Randomized Allocation with nonparametric estimation for a multi-armed bandit problem with covariates
Yang, Y., & Zhu, D. (2002) · The Annals of Statistics · doi:10.1214/aos/1015362186 · Cited by 83
Incomplete Learning from Endogenous Data in Dynamic Allocation
Brezzi, M., & Lai, T. L. (2000) · Econometrica · doi:10.1111/1468-0262.00170 · Cited by 44
Cited by
Machine Learning Methods That Economists Should Know About
Athey, S., & Imbens, G. W. (2019) · Annual Review of Economics · doi:10.1146/annurev-economics-080217-053433 · Cited by 1,163
(2011) · Cited by 1,000
Analysis of Thompson Sampling for the multi-armed bandit problem
Agrawal, S., & Goyal, N. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.1797 · Cited by 819
Online learning: A comprehensive survey
Hoi, S. C. H., Sahoo, D., Lu, J., & Zhao, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.112 · Cited by 746
Thompson Sampling for Contextual Bandits with Linear Payoffs
Agrawal, S., & Goyal, N. (2012) · arXiv · doi:10.48550/arxiv.1209.3352 · Cited by 600
Metadata
- Authors
- Scott, S. L.
- Venue
- Applied Stochastic Models in Business and Industry
- Publisher
- Wiley
- Published
- 2010-11
- Volume
- 26
- Issue
- 6
- Pages
- 639-658
- ISSN
- 1524-1904 (ISSN-L)
- 1526-4025
- OpenAlex
- W2158125716
- Cited by
- 406
- References
- 27
- FWCI
- 13.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.