The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
Abstract
This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rockafellar, R. T. (1970) · Princeton University Press eBooks · doi:10.1515/9781400873173 · Cited by 10,801
Doob, J. L., & Doob, J. L. (1953) · CERN Document Server (European Organization for Nuclear Research) · Cited by 9,053
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316887 · Cited by 6,024
Adaptive Control Processes: A Guided Tour
Bellman, R. E. (1961) · doi:10.1515/9781400874668 · Cited by 2,802
Probability Measures on Metric Spaces.
Rao, B. L. S. P., & Parthasarathy, K. R. (1968) · Journal of the American Statistical Association · doi:10.2307/2283907 · Cited by 2,424
Shapley, L. S. (1953) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.39.10.1095 · Cited by 2,382
Stochastic optimal control : the discrete time case
Bertsekas, D. P., & Shreve, S. E. (2007) · Cited by 1,654
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
Smallwood, R. D., & Sondik, E. J. (1973) · Operations Research · doi:10.1287/opre.21.5.1071 · Cited by 1,479
Stochastic Stability and Control
(1967) · Mathematics in Science and Engineering · doi:10.1016/s0076-5392(08)x6039-4 · Cited by 1,452
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,325
Introduction to Stochastic Dynamic Programming.
Berry, D. A., & Ross, S. M. (1986) · Journal of the American Statistical Association · doi:10.2307/2289262 · Cited by 1,133
Stochastic systems : estimation, identification, and adaptive control
Kumar, P. R., & Varaiya, P. (1986) · Medical Entomology and Zoology · Cited by 1,018
Arrow, K. J., Harris, T., & Marschak, J. (1951) · Econometrica · doi:10.2307/1906813 · Cited by 967
Discounted Dynamic Programming
Blackwell, D. (1965) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177700285 · Cited by 918
Optimal control of Markov processes with incomplete state information
Åström, K. J. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90154-x · Cited by 854
Finite State Markovian Decision Processes
Derman, C. (1970) · Cited by 853
Monahan, G. E. (1982) · Management Science · doi:10.1287/mnsc.28.1.1 · Cited by 745
Dynamic Programming and Markov Processes.
Kaufman, H., & Howard, R. A. (1961) · The American Mathematical Monthly · doi:10.2307/2312519 · Cited by 638
Loève, M. (1978) · Graduate Texts in Mathematics · doi:10.1007/978-1-4612-6257-2 · Cited by 545
Sondik, E. J. (1978) · Operations Research · doi:10.1287/opre.26.2.282 · Cited by 536
A survey of algorithmic methods for partially observed Markov decision processes
Lovejoy, W. S. (1991) · Annals of Operations Research · doi:10.1007/bf02055574 · Cited by 532
Statistical Decision Functions.
Lindley, D. V., & Wald, A. (1951) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2980980 · Cited by 475
The optimal control of partially observable Markov processes
Sondik, E. J. (1971) · Defense Technical Information Center (DTIC) · Cited by 440
Linear Programming and Sequential Decisions
Manne, A. S. (1960) · Management Science · doi:10.1287/mnsc.6.3.259 · Cited by 437
Chapter 8 Markov decision processes
Puterman, M. L. (1990) · Handbooks in operations research and management science · doi:10.1016/s0927-0507(05)80172-0 · Cited by 421
White, C. C., & White, D. J. (1989) · European Journal of Operational Research · doi:10.1016/0377-2217(89)90348-2 · Cited by 415
Discrete Dynamic Programming with Sensitive Discount Optimality Criteria
Veinott, A. F. (1969) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697379 · Cited by 248
A Survey of Some Results in Stochastic Adaptive Control
Kumar, P. R. (1985) · SIAM Journal on Control and Optimization · doi:10.1137/0323023 · Cited by 217
Technical Note—An Equivalence Between Continuous and Discrete Time Markov Decision Processes
Serfozo, R. F. (1979) · Operations Research · doi:10.1287/opre.27.3.616 · Cited by 212
9. STOCHASTIC GAMES WITH ZERO STOP PROBABILITIES
Gillette, D. (1958) · Contributions to the Theory of Games (AM-39), Volume III · doi:10.1515/9781400882151-011 · Cited by 204
Sufficient statistics in the optimum control of stochastic systems
Striebel, C. (1965) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(65)90027-2 · Cited by 180
An Equivalence between Continuous and Discrete Time Markov Decision Processes.
Serfozo, R. F. (1976) · Defense Technical Information Center (DTIC) · Cited by 159
Linear Programming and Markov Decision Chains
Hordijk, A., & Kallenberg, L. C. M. (1979) · Management Science · doi:10.1287/mnsc.25.4.352 · Cited by 143
Markovian Sequential Replacement Processes
Taylor, H. M. (1965) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177699796 · Cited by 136
Some Monotonicity Results for Partially Observed Markov Decision Processes
Lovejoy, W. S. (1987) · Operations Research · doi:10.1287/opre.35.5.736 · Cited by 135
Mean, variance, and probabilistic criteria in finite Markov decision processes: A review
White, D. J. (1988) · Journal of Optimization Theory and Applications · doi:10.1007/bf00938524 · Cited by 118
A Quality Control Model with Learning Effects
Fine, C. H. (1988) · Operations Research · doi:10.1287/opre.36.3.437 · Cited by 104
Arbitrary State Markovian Decision Processes
Ross, S. M. (1968) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177698041 · Cited by 102
Measurable selection theorems for optimization problems
Rieder, U. (1978) · manuscripta mathematica · doi:10.1007/bf01168566 · Cited by 73
Optimal Inspection and Repair of a Production Process Subject to Deterioration
White, C. C. (1978) · Journal of the Operational Research Society · doi:10.1057/jors.1978.51 · Cited by 67
Åström, K. J. (1969) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(69)90163-2 · Cited by 63
The structure of dynamic programing models
Karlin, S. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020408 · Cited by 62
Optimal Infinite-Horizon Undiscounted Control of Finite Probabilistic Systems
Platzman, L. K. (1980) · SIAM Journal on Control and Optimization · doi:10.1137/0318028 · Cited by 61
Optimal Control of Discrete Time Stochastic Systems
Striebel, C. T. (1975) · Lecture notes in economics and mathematical systems · doi:10.1007/978-3-642-45470-7 · Cited by 60
Fernández-Gaucherand, E., Arapostathis, A., & Marcus, S. I. (1991) · Annals of Operations Research · doi:10.1007/bf02283610 · Cited by 59
Self-citationDenardo, E. V. (1973) · Mathematical Programming · doi:10.1016/b978-0-12-358350-5.50005-1 · Cited by 30
Monotone control laws for noisy, countable-state Markov chains
White, C. C. (1980) · European Journal of Operational Research · doi:10.1016/0377-2217(80)90211-8 · Cited by 29
Bounds on optimal cost for a replacement problem with partial observations
White, C. C. (1979) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800260305 · Cited by 27
Finite Memory Estimation and Control of Finite Probabilistic Systems.
Platzman, L. K. (1977) · Cited by 25
Fernández-Gaucherand, E., Arapostathis, A., & Marcus, S. I. (1990) · Systems & Control Letters · doi:10.1016/0167-6911(90)90067-5 · Cited by 24
Self-citationOn partially observable Markov decision processes with an average cost criterion
Fernández-Gaucherand, E., Arapostathis, A., & Marcus, S. I. (2003) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1989.70341 · Cited by 11
Self-citation
Cited by
Constrained Markov Decision Processes: Stochastic Modeling
Altman, E. (2021) · Routledge eBooks · doi:10.1201/9781315140223 · Cited by 1,693
Lattimore, T., & Szepesvári, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108571401 · Cited by 976
Optimality of Myopic Sensing in Multichannel Opportunistic Access
Ahmad, S. H. A., Liu, M., Javidi, T., Zhao, Q., & Krishnamachari, B. (2009) · IEEE Transactions on Information Theory · doi:10.1109/tit.2009.2025561 · Cited by 364
Liu, K., & Zhao, Q. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2068950 · Cited by 347
Reinforcement Learning: A Tutorial Survey and Recent Advances
Gosavi, A. (2009) · INFORMS Journal on Computing · doi:10.1287/ijoc.1080.0305 · Cited by 326
Metadata
- Authors
- Arapostathis, A., Borkar, V. S., Fernández-Gaucherand, E., Ghosh, M. K., & Marcus, S. I.
- Venue
- SIAM Journal on Control and Optimization
- Published
- 1993-03
- Volume
- 31
- Issue
- 2
- Pages
- 282-344
- ISSN
- 0363-0129 (ISSN-L)
- 1095-7138
- OpenAlex
- W1967037758
- Cited by
- 592
- References
- 51
- FWCI
- 12.04
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.