An analytic solution to discrete Bayesian reinforcement learning
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) was originally proposed as a framework to allow agents to learn in an online fashion as they interact with their environment. Existing RL algorithms come short of achieving this goal because the amount of exploration required is often too costly and/or too time consuming for online learning. As a result, RL is mostly used for offline learning in simulated environments. We propose a new algorithm, called BEETLE, for effective online learning that is computationally efficient while minimizing the amount of exploration. We take a Bayesian model-based approach, framing RL as a partially observable Markov decision process. Our two main contributions are the analytical derivation that the optimal value function is the upper envelope of a set of multivariate polynomials, and an efficient point-based value iteration algorithm that exploits this simple parameterization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Barto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R. D., & Sondik, E. J. (1973) · Operations Research · doi:10.1287/opre.21.5.1071 · Cited by 1,479
Temporal Difference Learning and TD-Gammon
Tesauro, G. (1995) · ICGA Journal · doi:10.3233/icg-1995-18207 · Cited by 764
Kaelbling, L. P. (1993) · doi:10.7551/mitpress/4168.001.0001 · Cited by 599
Improving Elevator Performance Using Reinforcement Learning
Crites, R. H., & Barto, A. G. (1995) · neural information processing systems · Cited by 588
Autonomous Inverted Helicopter Flight via Reinforcement Learning
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2006) · Springer Tracts in Advanced Robotics · doi:10.1007/11552246_35 · Cited by 553
Dearden, R., Friedman, N., & Russell, S. (1998) · National Conference on Artificial Intelligence · Cited by 551
Perseus: Randomized point-based value iteration for POMDPs
Spaan, M. T. J., & Vlassis, N. A. (2005) · UvA-DARE (University of Amsterdam) · Cited by 526
Self-citationAutonomous Helicopter Flight via Reinforcement Learning
Kim, H. J., Jordan, M. I., Sastry, S., & Ng, A. Y. (2003) · neural information processing systems · Cited by 390
Robotics: Science and Systems V
(2009) · doi:10.15607/rss.2009.v · Cited by 367
A Bayesian Framework for Reinforcement Learning
Strens, M. J. A. (2000) · International Conference on Machine Learning · Cited by 358
Optimal learning: computational procedures for bayes-adaptive markov decision processes
Duff, M. O., & Barto, A. G. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 239
Model-Based Bayesian Exploration
Dearden, R., Friedman, N., & André, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6690 · Cited by 221
(2005) · Discovery Research Portal (University of Dundee) · Cited by 158
Bayesian sparse sampling for on-line reward optimization
Wang, T., Lizotte, D. J., Bowling, M., & Schuurmans, D. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102472 · Cited by 142
Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty
Meuleau, N., & Bourgine, P. (1999) · Machine Learning · doi:10.1023/a:1007541107674 · Cited by 110
Robot Planning in Partially Observable Continuous Domains
Porta, J. M., Spaan, M. T. J., & Vlassis, N. A. (2005) · Robotics: Science and Systems · doi:10.15607/rss.2005.i.029 · Cited by 50
Self-citationDuff, M. J. B. (2003) · UCL Discovery (University College London) · Cited by 48
Cited by
A Roadmap of Agent Research and Development
Jennings, N. R., Sycara, K., & Wooldridge, M. (1998) · Autonomous Agents and Multi-Agent Systems · doi:10.1023/a:1010090405266 · Cited by 2,161
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
“L G ] 1 5 M ay 2 01 7 et al., 2012; Poupart et al., 2006) or, 2) encourage the agent to perform actions that reduce the error/uncertainty in the agent’s ability to predict the consequence of its own actions (i.e. its knowledge about the environment) (Houthooft et al., 2016; Mohamed & Rezende, 2015;…”
BackgroundAlgorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Machine learning in medical applications: a review of state-of-the-art methods
Shehab, M., Abualigah, L., Shambour, Q., Abu-Hashem, M. A., Shambour, M. K. Y., Alsalibi, A. I., & Gandomi, A. H. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105458 · Cited by 542
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401
“Examples of intrinsic reward include “curiosity” [11, 22, 27, 35, 40] which uses prediction error as reward signal, and “visitation counts” [3, 20, 24, 30] which discourages the agent from revisiting the same states.”
Background
Metadata
- Authors
- Poupart, P., Vlassis, N., Hoey, J., & Regan, K.
- Venue
- Proceedings of the 23rd international conference on Machine learning - ICML '06
- Publisher
- ACM Press
- Published
- 2006
- Pages
- 697-704
- OpenAlex
- W2116459397
- Cited by
- 278
- References
- 20
- FWCI
- 9.84
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.