An analysis of model-based Interval Estimation for Markov Decision Processes
Open accessIntegrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RiverSwim tests deep exploration under stochastic transitions: the small reward in s0 is available immediately and yields a return of 1000 per episode, whereas the large reward requires swimming against the current for many consecutive steps without intermediate feedback, so an agent that dithers around the locally optimal 'always LEFT' policy is carried back by the current and almost never ReACHs
“RiverSwim tests deep exploration under stochastic transitions. The small reward in s 0 is available immediately and yields a return of 1000 per episode, whereas the large reward requires swimming against the current for many consecutive steps without any intermediate feedback. An agent that dithers around the locally optimal 'always LEFT' policy is carried back by the current and almost never ReAC”
From Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learning (2026) · cites this paper as Strehl & Littman, 2008
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
Valiant, L. G. (1984) · Communications of the ACM · doi:10.1145/1968.1972 · Cited by 3,279 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · see citing works
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527 · see citing works
Emergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424 · see citing works
“One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
BackgroundMethodMachado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
Reinforcement learning for HVAC control in intelligent buildings: A technical and conceptual review
Al Sayed, K., Boodi, A., Sadeghian Broujeny, R., & Beddiar, K. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.110085 · Cited by 140 · see citing works
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129 · see citing works
An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey
Aubret, A., Matignon, L., & Hassas, S. (2023) · Entropy · doi:10.3390/e25020327 · Cited by 52 · see citing works
Count-based exploration with the successor representation
Machado, M. C., Bellemare, M. G., & Bowling, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5955 · Cited by 19 · see citing works
A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1 · see citing works
“These include model-free methods such as RND (Burda et al., 2019b), ICM (Pathak et al., 2017a) and pseudocounts (Bellemare et al., 2016b; Strehl & Littman, 2008; Bellemare et al., 2016a; Ostrovski et al., 2017; Martin et al., 2017; Tang et al., 2017; Machado et al., 2020), as well as model-based…”
MethodSpace-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learning
Störck, F., Hartmann, P., Hinder, F., Neumann, K., & Hammer, B. (2026) · arXiv · doi:10.48550/arxiv.2606.11797
- Authors
- Strehl, A. L., & Littman, M. L.
- Affiliations
- Yahoo (United States) (US)
- Rutgers, The State University of New Jersey (US)
- Corresponding author
- Littman, Michael L.
- ORCID
- Littman, Michael L.
- Venue
- Journal of Computer and System Sciences
- Publisher
- Elsevier BV
- Published
- 2008-12
- Volume
- 74
- Issue
- 8
- Pages
- 1309-1331
- ISSN
- 0022-0000 (ISSN-L)
- OpenAlex
- W1988526405
- Cited by
- 428
- References
- 3
- Citations by year
- 2026: 12 · 2025: 14 · 2024: 19 · 2023: 38 · 2022: 30 · 2021: 50 · 2020: 106 · 2019: 39 · 2018: 28 · 2017: 15 · 2016: 8 · 2015: 8 · 2014: 6 · 2013: 12 · 2012: 12
- FWCI
- 14.80
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Computer and System Sciences (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- Markov decision process, reinforcement learning, policy learning, exploration-exploitation balance, worst-case analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
