The DOI matches its Crossref record.
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Abstract
Preface. Acknowledgments. 1. The challenges of dynamic programming. 1.1 A dynamic programming example: a shortest path problem. 1.2 The three curses of dimensionality. 1.3 Some real applications. 1.4 Problem classes. 1.5 The many dialects of dynamic programming. 1.6 What is new in this book? 1.7 Bibliographic notes. 2. Some illustrative models. 2.1 Deterministic problems. 2.2 Stochastic problems. 2.3 Information acquisition problems. 2.4 A simple modeling framework for dynamic programs. 2.5 Bibliographic notes. Problems. 3. Introduction to Markov decision processes. 3.1 The optimality equations. 3.2 Finite horizon problems. 3.3 Infinite horizon problems. 3.4 Value iteration. 3.5 Policy iteration. 3.6 Hybrid valuepolicy iteration. 3.7 The linear programming method for dynamic programs. 3.8 Monotone policies. 3.9 Why does it work? 3.10 Bibliographic notes. Problems 4. Introduction to approximate dynamic programming. 4.1 The three curses of dimensionality (revisited). 4.2 The basic idea. 4.3 Sampling random variables . 4.4 ADP using the postdecision state variable. 4.5 Lowdimensional representations of value functions. 4.6 So just what is approximate dynamic programming? 4.7 Experimental issues. 4.8 Dynamic programming with missing or incomplete models. 4.9 Relationship to reinforcement learning. 4.10 But does it work? 4.11 Bibliographic notes. Problems. 5. Modeling dynamic programs. 5.1 Notational style. 5.2 Modeling time. 5.3 Modeling resources. 5.4 The states of our system. 5.5 Modeling decisions. 5.6 The exogenous information process. 5.7 The transition function. 5.8 The contribution function. 5.9 The objective function. 5.10 A measuretheoretic view of information. 5.11 Bibliographic notes. Problems. 6. Stochastic approximation methods. 6.1 A stochastic gradient algorithm. 6.2 Some stepsize recipes. 6.3 Stochastic stepsizes. 6.4 Computing bias and variance. 6.5 Optimal stepsizes. 6.6 Some experimental comparisons of stepsize formulas. 6.7 Convergence. 6.8 Why does it work? 6.9 Bibliographic notes. Problems. 7. Approximating value functions. 7.1 Approximation using aggregation. 7.2 Approximation methods using regression models. 7.3 Recursive methods for regression models. 7.4 Neural networks. 7.5 Batch processes. 7.6 Why does it work? 7.7 Bibliographic notes. Problems. 8. ADP for finite horizon problems. 8.1 Strategies for finite horizon problems. 8.2 Qlearning. 8.3 Temporal difference learning. 8.4 Policy iteration. 8.5 Monte Carlo value and policy iteration. 8.6 The actorcritic paradigm. 8.7 Bias in value function estimation. 8.8 State sampling strategies. 8.9 Starting and stopping. 8.10 A taxonomy of approximate dynamic programming strategies. 8.11 Why does it work? 8.12 Bibliographic notes. Problems. 9. Infinite horizon problems. 9.1 From finite to infinite horizon. 9.2 Algorithmic strategies. 9.3 Stepsizes for infinite horizon problems. 9.4 Error measures. 9.5 Direct ADP for online applications. 9.6 Finite horizon models for steady state applications. 9.7 Why does it work? 9.8 Bibliographic notes. Problems. 10. Exploration vs. exploitation. 10.1 A learning exercise: the nomadic trucker. 10.2 Learning strategies. 10.3 A simple information acquisition problem. 10.4 Gittins indices and the information acquisition problem. 10.5 Variations. 10.6 The knowledge gradient algorithm. 10.7 Information acquisition in dynamic programming. 10.8 Bibliographic notes. Problems. 11. Value function approximations for special functions. 11.1 Value functions versus gradients. 11.2 Linear approximations. 11.3 Piecewise linear approximations. 11.4 The SHAPE algorithm. 11.5 Regression methods. 11.6 Cutting planes. 11.7 Why does it work? 11.8 Bibliographic notes. Problems. 12. Dynamic resource allocation. 12.1 An asset acquisition problem. 12.2 The blood management problem. 12.3 A portfolio optimization problem. 12.4 A general resource allocation problem. 12.5 A fleet management problem. 12.6 A driver management problem. 12.7 Bibliographic references. Problems. 13. Implementation challenges. 13.1 Will ADP work for your problem? 13.2 Designing an ADP algorithm for complex problems. 13.3 Debugging an ADP algorithm. 13.4 Convergence issues. 13.5 Modeling your problem. 13.6 Online vs. offline models. 13.7 If it works, patent it!
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Solving high-dimensional partial differential equations using deep learning
Han, J., Jentzen, A., & E, W. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1718942115 · Cited by 1,722
Reinforcement learning and adaptive dynamic programming for feedback control
Lewis, F. L., & Vrabie, D. (2009) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2009.933854 · Cited by 1,558
Fast Model Predictive Control Using Online Optimization
Wang, Y., & Boyd, S. (2010) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2009.2017934 · Cited by 1,372
Neural Adaptive Video Streaming with Pensieve
Mao, H., Netravali, R., & Alizadeh, M. (2017) · Proceedings of the Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3098822.3098843 · Cited by 1,357
A review of dynamic vehicle routing problems
Pillac, V., Gendreau, M., Guéret, C., & Medaglia, A. L. (2013) · European Journal of Operational Research · doi:10.1016/j.ejor.2012.08.015 · Cited by 1,253
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Adaptive Dynamic Programming: An Introduction
Wang, F.-Y., Zhang, H., & Liu, D. (2009) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2009.932261 · Cited by 898
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Multi-agent Reinforcement Learning: An Overview
Buşoniu, L., Babuška, R., & De Schutter, B. (2010) · Studies in Computational Intelligence · doi:10.1007/978-3-642-14435-6_7 · Cited by 741
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 708
Modares, H., & Lewis, F. L. (2014) · Automatica · doi:10.1016/j.automatica.2014.05.011 · Cited by 701
Stochastic Optimization for Unit Commitment—A Review
Zheng, Q. P., Wang, J., & Liu, A. L. (2015) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2014.2355204 · Cited by 643
Kiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Naghibi-Sistani, M.-B. (2014) · Automatica · doi:10.1016/j.automatica.2014.02.015 · Cited by 574
Modares, H., Lewis, F. L., & Naghibi-Sistani, M.-B. (2014) · Automatica · doi:10.1016/j.automatica.2013.09.043 · Cited by 570
Lewis, F. L., & Vamvoudakis, K. G. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2043839 · Cited by 544
Rifaioglu, A. S., Atas, H., Martin, M. J., Cetin-Atalay, R., Atalay, V., & Doğan, T. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bby061 · Cited by 529
Vamvoudakis, K. G., & Lewis, F. L. (2011) · Automatica · doi:10.1016/j.automatica.2011.03.005 · Cited by 525
Reinforcement learning for control: Performance, stability, and deep approximators
Buşoniu, L., de Bruin, T., Tolić, D., Kober, J., & Palunko, I. (2018) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2018.09.005 · Cited by 463
Proceedings of the 2002 Winter Simulation Conference
(2002) · Proceedings of the Winter Simulation Conference · doi:10.1109/wsc.2002.1166355 · Cited by 456
Modares, H., & Lewis, F. L. (2014) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2317301 · Cited by 449
Adaptive Dynamic Programming and Adaptive Optimal Output Regulation of Linear Systems
Gao, W., & Jiang, Z.-P. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2548662 · Cited by 413
A modern Bayesian look at the multi‐armed bandit
Scott, S. L. (2010) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.874 · Cited by 406
Peng, M., Sun, Y., Li, X., Mao, Z., & Wang, C. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2016.2548658 · Cited by 386
Off-Policy Reinforcement Learning for $ H_\infty $ Control Design
Luo, B., Wu, H.-N., & Huang, T. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2319577 · Cited by 362
A Tutorial on Bayesian Optimization
Frazier, P. I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02811 · Cited by 342
Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning
Ruelens, F., Claessens, B. J., Vandael, S., De Schutter, B., Babuska, R., & Belmans, R. (2017) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2016.2517211 · Cited by 333
Model-Free Optimal Tracking Control via Critic-Only Q-Learning
Luo, B., Liu, D., Huang, T., & Wang, D. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2585520 · Cited by 332
A unified framework for stochastic optimization
Powell, W. B. (2019) · European Journal of Operational Research · doi:10.1016/j.ejor.2018.07.014 · Cited by 314
Self-citationRabbi, M., Pfammatter, A., Zhang, M., Spring, B., & Choudhury, T. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.4160 · Cited by 301
Stochastic Optimization of Economic Dispatch for Microgrid Based on Approximate Dynamic Programming
Shuai, H., Fang, J., Ai, X., Tang, Y., Wen, J., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2798039 · Cited by 294
A Third Wave in the Economics of Climate Change
Farmer, J. D., Hepburn, C., Mealy, P., & Teytelboym, A. (2015) · Environmental and Resource Economics · doi:10.1007/s10640-015-9965-2 · Cited by 283
Approximate Dynamic Programming for Ambulance Redeployment
Maxwell, M. S., Restrepo, M., Henderson, S. G., & Topaloglu, H. (2010) · INFORMS Journal on Computing · doi:10.1287/ijoc.1090.0345 · Cited by 283
Approximate policy iteration: a survey and some new methods
Bertsekas, D. P. (2011) · Journal of Control Theory and Applications · doi:10.1007/s11768-011-1005-3 · Cited by 273
Markov decision processes: a tool for sequential decision making under uncertainty
Alagoz, O., Hsu, H., Schaefer, A. J., & Roberts, M. S. (2010) · Medical Decision Making · doi:10.1177/0272989x09353194 · Cited by 266
Customer Acquisition via Display Advertising Using Multi-Armed Bandit Experiments
Schwartz, E. M., Bradlow, E. T., & Fader, P. S. (2017) · Marketing Science · doi:10.1287/mksc.2016.1023 · Cited by 260
Giuliani, M., Castelletti, A., Pianosi, F., Mason, E., & Reed, P. M. (2016) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)wr.1943-5452.0000570 · Cited by 257
Gao, W., Jiang, Y., Jiang, Z.-P., & Chai, T. (2016) · Automatica · doi:10.1016/j.automatica.2016.05.008 · Cited by 234
Online routing and battery reservations for electric vehicles with swappable batteries
Adler, J. D., & Mirchandani, P. B. (2014) · Transportation Research Part B Methodological · doi:10.1016/j.trb.2014.09.005 · Cited by 103
Artificial intelligence as structural estimation: Deep Blue, Bonanza, and AlphaGo
Igami, M. (2020) · The Econometrics Journal · doi:10.1093/ectj/utaa005 · Cited by 28
Metadata
- Authors
- Powell, W. B.
- Venue
- Wiley Series in Probability and Statistics
- Publisher
- Wiley
- Published
- 2011-08-04
- ISSN
- 1940-6347 (ISSN-L)
- OpenAlex
- W1601081659
- Cited by
- 2,297
- References
- 0
- FWCI
- 78.29
- Type
- book
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.