Constrained Markov Decision Processes: Stochastic Modeling
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
This book provides a unified approach for the study of constrained Markov decision processes with a finite state space and unbounded costs. Unlike the single controller case considered in many other books, the author considers a single controller with several objectives, such as minimizing delays and loss, probabilities, and maximization of throughputs. It is desirable to design a controller that minimizes one cost objective, subject to inequality constraints on other cost objectives. This framework describes dynamic decision problems arising frequently in many engineering fields. A thorough overview of these applications is presented in the introduction. The book is then divided into three sections that build upon each other.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Convergence of Probability Measures
Billingsley, P. (1999) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316962 · Cited by 13,910
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316887 · Cited by 6,024
Markov Chains and Stochastic Stability
Meyn, S. P., & Tweedie, R. L. (1993) · Springer eBooks · doi:10.1007/978-1-4471-3267-7 · Cited by 5,041
An Introduction to the Theory of Point Processes
Daley, D. J., & Vere-Jones, D. (2008) · Probability and Its Applications · doi:10.1007/978-0-387-49835-5 · Cited by 3,351
Shapley, L. S. (1953) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.39.10.1095 · Cited by 2,382
Sion, M. (1958) · Pacific Journal of Mathematics · doi:10.2140/pjm.1958.8.171 · Cited by 1,843
Finite State Markovian Decision Processes
Derman, C. (1970) · Cited by 956
11. Extensive Games and the Problem of Information
Kuhn, H. W. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-012 · Cited by 913
Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
Arapostathis, A., Borkar, V. S., Fernández-Gaucherand, E., Ghosh, M. K., & Marcus, S. I. (1993) · SIAM Journal on Control and Optimization · doi:10.1137/0331018 · Cited by 592
Cited by
Stochastic Network Optimization with Application to Communication and Queueing Systems
Neely, M. J. (2010) · Synthesis lectures on communication networks · doi:10.2200/s00271ed1v01y201006cnt007 · Cited by 2,218
Update or Wait: How to Keep Your Data Fresh
Sun, Y., Uysal-Biyikoglu, E., Yates, R. D., Koksal, C. E., & Shroff, N. B. (2017) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2735804 · Cited by 817
Constrained Policy Optimization
Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017) · arXiv · doi:10.48550/arxiv.1705.10528 · Cited by 804
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., & Schoellig, A. P. (2022) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-042920-020211 · Cited by 769
Delay-optimal computation task scheduling for mobile-edge computing systems
Liu, J., Mao, Y., Zhang, J., & Letaief, K. B. (2016) · 2016 IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2016.7541539 · Cited by 759
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
“In our locomotion projects, we formulated a C-MDP that has inequality constraints on the roll and the pitch of the robot base, which constitutes a rough measure of balance.”
Two formal analyses of attack graphs
Jha, S., Sheyner, O., & Wing, J. (2003) · Proceedings 15th IEEE Computer Security Foundations Workshop. CSFW-15 · doi:10.1109/csfw.2002.1021806 · Cited by 494
Chen, X., Qu, G., Tang, Y., Low, S., & Li, N. (2022) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2022.3154718 · Cited by 415
She, C., Sun, C., Gu, Z., Li, Y., Yang, C., Poor, H. V., & Vucetic, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3053601 · Cited by 405
Opportunistic Spectrum Access via Periodic Channel Sensing
Zhao, Q., Geirhofer, S., Tong, L., & Sadler, B. M. (2008) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2007.907867 · Cited by 372
Zhang, T., Chen, W., Han, Z., & Cao, Z. (2014) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2013.2295591 · Cited by 296
Cognitive Medium Access: Constraining Interference Based on Experimental Models
Geirhofer, S., Tong, L., & Sadler, B. M. (2008) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2008.080109 · Cited by 261
Decision Making in Multiagent Systems: A Survey
Rizk, Y., Awad, M., & Tunstel, E. W. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2840971 · Cited by 219
Safe Exploration in Markov Decision Processes
Moldovan, T. M., & Abbeel, P. (2012) · arXiv · doi:10.48550/arxiv.1205.4810 · Cited by 152
“In general, it is NP-hard to decide whether there exist δ -safe policies with respect to a home state, s 0 , and a stopping time, T , for some belief, β .”
BackgroundAn empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
“Recent work in RL safety [Dalal et al. 2018; Achiam et al. 2017] has cast safety in the context of Constrained MDPs (CMDPs) [Altman 1999], and we will concentrate on pre-defined constraints on the environment in this context.”
BackgroundA Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11
Avoiding Side Effects in Complex Environments
Turner, A. M., Ratzlaff, N., & Tadepalli, P. (2020) · arXiv · doi:10.48550/arxiv.2006.06547 · Cited by 11
Parsafar, P. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-024-06510-1 · Cited by 9
Reinforcement Learning with Trajectory Feedback
Efroni, Y., Merlis, N., & Mannor, S. (2020) · arXiv · doi:10.48550/arxiv.2008.06036 · Cited by 3
Lodhiya, H., McManus, A., & Walker, R. (2026) · arXiv · doi:10.48550/arxiv.2609.27844
Ethics-Aware Safe Reinforcement Learning for Rare-Event Risk Control in Interactive Urban Driving
Li, D., & Okhrin, O. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3708405
Metadata
- Authors
- Altman, E.
- Venue
- Routledge eBooks
- Publisher
- Routledge
- Published
- 2021-12-13
- OpenAlex
- W1518931405
- Cited by
- 1,693
- References
- 26
- FWCI
- 101.30
- Type
- book
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.