The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Simulation-based optimization of Markov reward processes
Open accessAbstract
This paper proposes a simulation-based algorithm for optimizing the average reward in a finite-state Markov reward process that depends on a set of parameters. As a special case, the method applies to Markov decision processes where optimization takes place within a parametrized set of policies. The algorithm relies on the regenerative structure of finite-state Markov processes, involves the simulation of a single sample path, and can be implemented online. A convergence result (with probability 1) is provided.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
(1995) · Cited by 10,558
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316887 · Cited by 6,024
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
(1999) · Cited by 5,147
Wiley Series in Probability and Statistics
(2008) · Wiley series in probability and statistics · doi:10.1002/series1345 · Cited by 4,217
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 2,202
Self-citationAnalysis of recursive stochastic algorithms
Ljung, L. (1977) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1977.1101561 · Cited by 1,467
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_588 · Cited by 950
Practical issues in temporal difference learning
Tesauro, G. (1992) · Machine Learning · doi:10.1007/bf00992697 · Cited by 787
Gradient Estimation via Perturbation Analysis
(1991) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(91)90124-l · Cited by 707
Perturbation Analysis of Discrete Event Dynamic Systems
Ho, Y., & Cao, X. (1991) · Springer eBooks · doi:10.1007/978-1-4615-4024-3 · Cited by 692
Importance Sampling for Stochastic Simulations
Glynn, P. W., & Iglehart, D. L. (1989) · Management Science · doi:10.1287/mnsc.35.11.1367 · Cited by 635
Stochastic approximation with two time scales
Borkar, V. S. (1997) · Systems & Control Letters · doi:10.1016/s0167-6911(97)90015-3 · Cited by 389
Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems
Jaakkola, T., Singh, S. P., & Jordan, M. I. (1994) · Cited by 361
Actor-Critic--Type Learning Algorithms for Markov Decision Processes
Konda, V. R., & Borkar, V. S. (1999) · SIAM Journal on Control and Optimization · doi:10.1137/s036301299731669x · Cited by 234
Likelilood ratio gradient estimation: an overview
Glynn, P. W. (1987) · Proceedings of the 19th conference on Winter simulation - WSC '87 · doi:10.1145/318371.318612 · Cited by 219
Perturbation realization, potentials, and sensitivity analysis of Markov processes
Cao, X., & Chen, H.-F. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.633827 · Cited by 208
General results on the convergence of stochastic algorithms
Delyon, B. (1996) · IEEE Transactions on Automatic Control · doi:10.1109/9.536495 · Cited by 92
Gradient convergence in gradient methods
Bertsekas, D. P. (1997) · Cited by 40
Missing and noisy data in nonlinear time-series prediction
Tresp, V., & Hofmann, R. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1995.514873 · Cited by 18
Cited by
Konda, V. R., & Tsitsiklis, J. N. (2003) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012901385691 · Cited by 709
Self-citationNatural actor–critic algorithms
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2009) · Automatica · doi:10.1016/j.automatica.2009.07.008 · Cited by 559
Learning to trade via direct reinforcement
Moody, J., & Saffell, M. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935097 · Cited by 535
Dynamic Programming and Suboptimal Control: A Survey from ADP to MPC*
Bertsekas, D. P. (2005) · European Journal of Control · doi:10.3166/ejc.11.310-334 · Cited by 366
Reinforcement Learning Through Modulation of Spike-Timing-Dependent Synaptic Plasticity
Florian, R. V. (2007) · Neural Computation · doi:10.1162/neco.2007.19.6.1468 · Cited by 358
Metadata
- Authors
- Marbach, P., & Tsitsiklis, J. N.
- Venue
- IEEE Transactions on Automatic Control
- Published
- 2001
- Volume
- 46
- Issue
- 2
- Pages
- 191-209
- ISSN
- 0018-9286 (ISSN-L)
- OpenAlex
- W2120465407
- Cited by
- 306
- References
- 25
- FWCI
- 28.79
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.