The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Linearly-solvable Markov decision problems
Abstract
We introduce a class of MPDs which greatly simplify Reinforcement Learning. They have discrete state spaces and continuous control spaces. The controls have the effect of rescaling the transition probabilities of an underlying Markov chain. A control cost penalizing KL divergence between controlled and uncontrolled transition probabilities makes the minimization problem convex, and allows an-alytical computation of the optimal controls given the optimal value function. An exponential transformation of the optimal value function makes the minimized Bellman equation linear. Apart from their theoretical signicance, the new MDPs enable efcient approximations to traditional MDPs. Shortest path problems are approximated to arbitrary precision with largest eigenvalue problems, yielding an O (n) algorithm. Accurate approximations to generic MDPs are obtained via continuous embedding reminiscent of LP relaxation in integer programming. Off-policy learning of the optimal value function is possible without need for state-action values; the new algorithm (Z-learning) outperforms Q-learning. This work was supported by NSF grant ECS0524761. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Schölkopf, B., & Smola, A. J. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/4175.001.0001 · Cited by 11,703
(1995) · Cited by 10,558
Chung, F. (1996) · CBMS Regional Conference Series in Mathematics · doi:10.1090/cbms/092 · Cited by 5,524
Cited by
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
Action and behavior: a free-energy formulation
Friston, K. J., Daunizeau, J., Kilner, J., & Kiebel, S. J. (2010) · Biological Cybernetics · doi:10.1007/s00422-010-0364-z · Cited by 866
“In computational motor control (Wolpert and Miall 1996; Todorov and Jordan 2002; Todorov 2006; Shadmehr and Krakauer 2008), it appears in the form of optimal control theory.”
BackgroundOffline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
Metadata
- Authors
- Todorov, E.
- Venue
- Advances in Neural Information Processing Systems 19
- Publisher
- The MIT Press
- Published
- 2007-09-07
- Pages
- 1369-1376
- OpenAlex
- W2145060720
- Cited by
- 334
- References
- 4
- FWCI
- 13.27
- Type
- book chapter
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Linearly-solvable Markov decision problems. https://app.cochat.ai/papers/10.7551/mitpress/7503.003.0176
- The paper
- Todorov, E. (2007). Linearly-solvable Markov decision problems. Advances in Neural Information Processing Systems 19. https://doi.org/10.7551/mitpress/7503.003.0176
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.