Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control
Open accessAbstract
Memory-limited partially observable stochastic control (ML-POSC) is the stochastic optimal control problem under incomplete information and memory limitation. To obtain the optimal control function of ML-POSC, a system of the forward Fokker–Planck (FP) equation and the backward Hamilton–Jacobi–Bellman (HJB) equation needs to be solved. In this work, we first show that the system of HJB-FP equations can be interpreted via Pontryagin’s minimum principle on the probability density function space. Based on this interpretation, we then propose the forward-backward sweep method (FBSM) for ML-POSC. FBSM is one of the most basic algorithms for Pontryagin’s minimum principle, which alternately computes the forward FP equation and the backward HJB equation in ML-POSC. Although the convergence of FBSM is generally not guaranteed in deterministic control and mean-field stochastic control, it is guaranteed in ML-POSC because the coupling of the HJB-FP equations is limited to the optimal control function in ML-POSC.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlled Markov Processes and Viscosity Solutions
Fleming, W. H., & Soner, H. M. (2005) · Springer eBooks · doi:10.1007/0-387-31071-1 · Cited by 2,794 · see citing works
Optimal Control Applied to Biological Models
Lenhart, S., & Workman, J. T. (2007) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420011418 · Cited by 2,167 · see citing works
Lewis, F. L., Vrabie, D. L., & Syrmos, V. L. (2012) · doi:10.1002/9781118122631 · Cited by 2,020 · see citing works
Yong, J., & Zhou, X. Y. (1999) · doi:10.1007/978-1-4612-1466-3 · Cited by 1,243 · see citing works
Mean Field Games and Mean Field Type Control Theory
Bensoussan, A., Frehse, J., & Yam, S. C. P. (2013) · SpringerBriefs in mathematics · doi:10.1007/978-1-4614-8508-7 · Cited by 789 · see citing works
Stochastic Control of Partially Observable Systems
Bensoussan, A. (1992) · Cambridge University Press eBooks · doi:10.1017/cbo9780511526503 · Cited by 674 · see citing works
Numerical Methods for Stochastic Control Problems in Continuous Time
Kushner, H. J., & Dupuis, P. G. (1992) · Springer eBooks · doi:10.1007/978-1-4684-0441-8 · Cited by 632 · see citing works
Probabilistic Theory of Mean Field Games with Applications II
Carmona, R. A., & Delarue, F. (2018) · Probability theory and stochastic modelling · doi:10.1007/978-3-319-56436-4 · Cited by 575 · see citing works
On the Separation Theorem of Stochastic Control
Wonham, W. M. (1968) · SIAM Journal on Control · doi:10.1137/0306023 · Cited by 547 · see citing works
Probabilistic Theory of Mean Field Games with Applications I
Carmona, R. A., & Delarue, F. (2018) · Probability theory and stochastic modelling · doi:10.1007/978-3-319-58920-6 · Cited by 516 · see citing works
Path integrals and symmetry breaking for optimal control theory
Kappen, H. J. (2005) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2005/11/p11011 · Cited by 309 · see citing works
Linear Theory for Control of Nonlinear Stochastic Systems
Kappen, H. J. (2005) · Physical Review Letters · doi:10.1103/physrevlett.95.200201 · Cited by 307 · see citing works
Dynamic Programming Conditions for Partially Observable Stochastic Systems
Davis, M. H., & Varaiya, P. P. (1973) · SIAM Journal on Control · doi:10.1137/0311020 · Cited by 231 · see citing works
Convergence of the forward-backward sweep method in optimal control
McAsey, M. J., Mou, L., & Han, W. (2012) · Computational Optimization and Applications · doi:10.1007/s10589-011-9454-7 · Cited by 214 · see citing works
Ruthotto, L., Osher, S., Li, W., Nurbekyan, L., & Fung, S. W. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1922204117 · Cited by 205 · see citing works
Vinter, R. (2010) · Birkhäuser Boston eBooks · doi:10.1007/978-0-8176-8086-2 · Cited by 195 · see citing works
Li, W., & Todorov, E. V. (2007) · International Journal of Control · doi:10.1080/00207170701364913 · Cited by 176 · see citing works
Kushner, H. J. (1962) · IRE Transactions on Automatic Control · doi:10.1109/tac.1962.1105490 · Cited by 172 · see citing works
Stochastic Optimal Control in Infinite Dimension
Fabbri, G., Gozzi, F., & Święch, A. (2017) · Probability theory and stochastic modelling · doi:10.1007/978-3-319-53067-3 · Cited by 167 · see citing works
A numerical method for solving parabolic equations with opposite orientations
Hackbusch, W. (1978) · Computing · doi:10.1007/bf02251947 · Cited by 145 · see citing works
The Master equation in mean field theory
Bensoussan, A., Frehse, J., & Yam, S. C. P. (2015) · Journal de Mathématiques Pures et Appliquées · doi:10.1016/j.matpur.2014.11.005 · Cited by 138 · see citing works
Bellman equation and viscosity solutions for mean-field stochastic control problem
Pham, H., & Wei, X. (2017) · ESAIM Control Optimisation and Calculus of Variations · doi:10.1051/cocv/2017019 · Cited by 126 · see citing works
The Master Equation for Large Population Equilibriums
Carmona, R. A., & Delarue, F. (2014) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-319-11292-3_4 · Cited by 118 · see citing works
Successive approximation methods for the solution of optimal control problems
Mitter, S. K. (1966) · Automatica · doi:10.1016/0005-1098(66)90009-4 · Cited by 108 · see citing works
A Fully Discrete Semi-Lagrangian Scheme for a First Order Mean Field Game Problem
Carlini, E., & Silva, F. J. (2014) · SIAM Journal on Numerical Analysis · doi:10.1137/120902987 · Cited by 103 · see citing works
Method of successive approximations for solution of optimal control problems
Chernousko, F. L., & Lyubushin, A. A. (1982) · Optimal Control Applications and Methods · doi:10.1002/oca.4660030201 · Cited by 96 · see citing works
On the interpretation of the Master Equation
Bensoussan, A., Frehse, J., & Yam, S. C. P. (2017) · Stochastic Processes and their Applications · doi:10.1016/j.spa.2016.10.004 · Cited by 76 · see citing works
Maximum Principle Based Algorithms for Deep LearningPreprint
Li, Q., Chen, L., Tai, C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.09513 · Cited by 76 · see citing works
Lin, A. T., Fung, S. W., Li, W., Nurbekyan, L., & Osher, S. (2020) · arXiv (Cornell University) · doi:10.1073/pnas.2024713118 · Cited by 72 · see citing works
Stochastic Control Theory: Dynamic Programming Principle
Nisio, M. (2015) · Probability Theory and Stochastic Modelling · doi:10.1007/978-4-431-55123-2 · Cited by 67 · see citing works
Dynamic programming for mean-field type control
Laurière, M., & Pironneau, O. (2014) · Comptes Rendus Mathématique · doi:10.1016/j.crma.2014.07.008 · Cited by 66 · see citing works
An Introduction to Optimal Control of FBSDE with Incomplete Information
Wang, G., Wu, Z., & Xiong, J. (2018) · SpringerBriefs in mathematics · doi:10.1007/978-3-319-79039-8 · Cited by 59 · see citing works
Numerical methods for mean field games and mean field type control
Laurière, M. (2021) · Proceedings of symposia in applied mathematics · doi:10.1090/psapm/078/06 · Cited by 57 · see citing works
Estimation and Control of Dynamical Systems
Bensoussan, A. (2018) · Interdisciplinary Applied Mathematics · doi:10.1007/978-3-319-75456-7 · Cited by 51 · see citing works
On a method of successive approximations for the solution of problems of optimal control
Krylov, I. A., & Chernousko, F. L. (1963) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(63)90353-7 · Cited by 38 · see citing works
An Iterative Optimal Control and Estimation Design for Nonlinear Stochastic System
Li, W., & Todorov, E. (2006) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.2006.377485 · Cited by 32 · see citing works
An Iterative Method for Nonlinear Stochastic Optimal Control Based on Path Integrals
Satoh, S., Kappen, H. J., & Saeki, M. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2547979 · Cited by 30 · see citing works
Implementation and acceleration of optimal control for systems biology
Sharp, J. A., Burrage, K., & Simpson, M. J. (2021) · Journal of The Royal Society Interface · doi:10.1098/rsif.2021.0241 · Cited by 29 · see citing works
Dynamic Programming for Mean-Field Type Control
Laurière, M., & Pironneau, O. (2015) · Journal of Optimization Theory and Applications · doi:10.1007/s10957-015-0785-x · Cited by 27 · see citing works
A policy iteration method for mean field games
Cacace, S., Camilli, F., & Goffi, A. (2021) · ESAIM Control Optimisation and Calculus of Variations · doi:10.1051/cocv/2021081 · Cited by 26 · see citing works
Liu, X., & Frank, J. (2020) · Journal of Computational and Applied Mathematics · doi:10.1016/j.cam.2020.113133 · Cited by 26 · see citing works
Connection between the Bacterial Chemotactic Network and Optimal Filtering
Nakamura, K., & Kobayashi, T. J. (2021) · Physical Review Letters · doi:10.1103/physrevlett.126.128102 · Cited by 19 · see citing works
Self-citationAschepkov, L. T., Dolgy, D. V., Kim, T., et al. (2016) · Springer eBooks · doi:10.1007/978-3-319-49781-5 · Cited by 19 · see citing works
Rates of convergence for the policy iteration method for Mean Field Games systems
Camilli, F., & Tang, Q. (2022) · Journal of Mathematical Analysis and Applications · doi:10.1016/j.jmaa.2022.126138 · Cited by 18 · see citing works
Chemotaxis emerges as the optimal solution to cooperative search games
Pezzotta, A., Adorisio, M., & Celani, A. (2018) · Physical Review E · doi:10.1103/physreve.98.042401 · Cited by 17 · see citing works
Optimal sensing and control of run-and-tumble chemotaxis
Nakamura, K., & Kobayashi, T. J. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.013120 · Cited by 17 · see citing works
Self-citationMinimum-information LQG control part I: Memoryless controllers
Fox, R., & Tishby, N. (2016) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.2016.7799131 · Cited by 16 · see citing works
Semi-Lagrangian schemes for mean field game models
Carlini, E., & Silva, F. J. (2013) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.2013.6760358 · Cited by 11 · see citing works
Minimum-information LQG control Part II: Retentive controllers
Fox, R., & Tishby, N. (2016) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.2016.7799130 · Cited by 11 · see citing works
Memory-Limited Partially Observable Stochastic Control and Its Mean-Field Control Approach
Tottori, T., & Kobayashi, T. J. (2022) · Entropy · doi:10.3390/e24111599 · Cited by 11 · see citing works
Self-citationMean field approach to stochastic control with partial information
Bensoussan, A., & Yam, S. C. P. (2021) · ESAIM Control Optimisation and Calculus of Variations · doi:10.1051/cocv/2021085 · Cited by 10 · see citing works
Optimal collision avoidance in swarms of active Brownian particles
Borra, F., Cencini, M., & Celani, A. (2021) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac12c6 · Cited by 10 · see citing works
A semi-Lagrangian scheme for a degenerate second order mean field game system
Carlini, E., & Silva, F. J. (2015) · Discrete and Continuous Dynamical Systems · doi:10.3934/dcds.2015.35.4269 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [21]Puterman M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming Wiley-Interscience Hoboken, NJ, USA 2014
- [22]Pontryagin L.S. Mathematical Theory of Optimal Processes CRC Press Boca Raton, FL, USA 1987
- [46]Bellman R. Dynamic Programming Princeton University Press Princeton, NJ, USA 1957
- [47]Howard R.A. Dynamic Programming and Markov Processes John Wiley Oxford, UK 1960
- [52]Laurière M. Song J. Tang Q. Policy iteration method for time-dependent Mean Field Games systems with non-separable Hamiltonians arXiv 2021 2110.02552 10.1007/s00245-022-09925-5 · DOI
Cited by
Sorted by citations
Schnyder, S. K., Molina, J. J., Yamamoto, R., & Turner, M. S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2409362122 · Cited by 12 · see citing works
Memory-Limited Partially Observable Stochastic Control and Its Mean-Field Control Approach
Tottori, T., & Kobayashi, T. J. (2022) · Entropy · doi:10.3390/e24111599 · Cited by 11 · see citing works
Self-citationDecentralized Stochastic Control with Finite-Dimensional Memories: A Memory Limitation Approach
Tottori, T., & Kobayashi, T. J. (2023) · Entropy · doi:10.3390/e25050791 · Cited by 7 · see citing works
Self-citationTottori, T., & Kobayashi, T. J. (2025) · Physical Review Research · doi:10.1103/gvl6-cvby · Cited by 6 · see citing works
Self-citationResource limitations induce phase transitions in biological information processing
Tottori, T., & Kobayashi, T. J. (2025) · Physical Review Research · doi:10.1103/nz4j-tyv1 · Cited by 3 · see citing works
Self-citationTheoretical Analysis of Resource-Induced Phase Transitions in Estimation Strategies
Tottori, T., & Kobayashi, T. J. (2026) · Physical Review Letters · doi:10.1103/5ynb-7k4v
Self-citation
- Authors
- Tottori, T., & Kobayashi, T. J.
- Affiliations
- The University of Tokyo (JP)
- Institute of Industrial Science, The University of Tokyo (JP)
- Corresponding author
- Tottori, Takehiro
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2023-01-21
- Volume
- 25
- Issue
- 2
- Pages
- 208
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC9955073
- OpenAlex
- W4317734540
- Cited by
- 11
- References
- 53
- Citations by year
- 2026: 1 · 2025: 6 · 2023: 3 · 2022: 1
- FWCI
- 1.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- JSPS Research Fellowship (21J20436, 19H05799, JPMJCR2011)
- JSPS KAKENHI (21J20436, 19H05799, JPMJCR2011)
- JST CREST (21J20436, 19H05799, JPMJCR2011)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Social Sciences › Economics, Econometrics and Finance › Finance › Stochastic processes and financial applications
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- forward-backward sweep method, Fokker-Planck equation, Hamilton-Jacobi-Bellman equation, Pontryagin maximum principle, incomplete information, stochastic optimal control, convergence analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
