Path integrals and symmetry breaking for optimal control theory
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton–Jacobi–Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton–Jacobi equation to the Schrödinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process that can be computed by stochastic integration or by the evaluation of a path integral. It is shown how in the deterministic limit the Pontryagin minimum principle formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as Monte Carlo sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in a number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and one can simulate active inference using generalisations of linear quadratic control or model predictive control [93, 94]:”
From The free energy principle made simpler but not too simple (2023) · cites this paper as [93]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Mathematical Theory of Optimal Processes.
Blum, E. K., Pontryagin, Boltyanskii, Gamkrelidze, & Mishchenko (1963) · The American Mathematical Monthly · doi:10.2307/2312867 · Cited by 4,261
The Mathematical Theory of Optimal Processes
Bellman, R., Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., Mishchenko, E. F., Trirogoff, K. N., & Neustadt, L. W. (1965) · Econometrica · doi:10.2307/1911909 · Cited by 4,211
Mathematical Theory of Optimal Processes
Pontryagin, L. S. (2018) · Routledge eBooks · doi:10.1201/9780203749319 · Cited by 3,713 · cited 5×
“First we discuss the small noise limit, where we can use the Laplace approximation to recover the PMP formalism for deterministic control.”
MethodInfluentialT RANSITION P ATH S AMPLING : Throwing Ropes Over Rough Mountain Passes, in the Dark
Bolhuis, P. G., Chandler, D., Dellago, C., & Geissler, P. L. (2002) · Annual Review of Physical Chemistry · doi:10.1146/annurev.physchem.53.082301.113146 · Cited by 1,972 · cited 1×
“A prominent method in this field is transition path sampling [16], which can be viewed as a Metropolis–Hasting sampling scheme in path space, where a”
MethodQuantentheorie in hydrodynamischer Form
Madelung, E. (1927) · Zeitschrift f�r Physik · doi:10.1007/bf01400372 · Cited by 1,892
Optimal Control and Estimation
Stengel, R. F. (1994) · Cited by 1,617
Dynamical Theories of Brownian Motion
Nelson, E. (1967) · doi:10.1515/9780691219615 · Cited by 1,251
Yong, J., & Zhou, X. Y. (1999) · doi:10.1007/978-1-4612-1466-3 · Cited by 1,243
Dynamical Theories of Brownian Motion
NELSON, E. (2020) · Princeton University Press eBooks · doi:10.2307/j.ctv15r57jg · Cited by 1,123 · cited 2×
“Nelson showed that these equations describe a stochastic dynamics in a force field given by the ∇J , where the noise is proportional to h̄ [8, 18].”
BackgroundShampine, L. F., Gladwell, I., & Thompson, S. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511615542 · Cited by 828 · cited 1×
“Solving 2n ordinary differential equations may be more efficient than solving the n-dimensional partial differential equation, using shooting methods (see for instance [7]), but may be unstable in some cases.”
Method
Cited by
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
“The idea of constraining policy gradient updates was explored earlier through the use of “natural gradient” updates [46] and also by means of the Kullback-Leibler (KL) divergence [48, 88].”
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
“The idea of constraining policy gradient updates was explored earlier through the use of “natural gradient” updates [46] and also by means of the Kullback-Leibler (KL) divergence [48, 88].”
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
“In recent years, general reinforcement learning has yielded three kinds of policy search approaches that have translated particularly well into the domain of robotics: (i) policy gradients approaches based on likelihood-ratio estimation (Sutton et al, 2000), (ii) policy updates inspired by expectation maximization (To…”
BackgroundSoft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994
“Optimization problems of this type have been explored in a number of prior works (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008).”
BackgroundRobotics: Science and Systems VIII
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9816.001.0001 · Cited by 1,028
Metadata
- Authors
- Kappen, H. J.
- Affiliations
- Radboud University Nijmegen (NL)
- Corresponding author
- Kappen, H J
- ORCID
- Kappen, H J
- Venue
- Journal of Statistical Mechanics Theory and Experiment
- Publisher
- IOP Publishing
- Published
- 2005-11-01
- Volume
- 2005
- Issue
- 11
- Pages
- P11011-P11011
- ISSN
- 1742-5468 (ISSN-L)
- OpenAlex
- W2026659355
- Cited by
- 309
- References
- 22
- Citations by year
- 2026: 12 · 2025: 24 · 2024: 14 · 2023: 23 · 2022: 19 · 2021: 27 · 2020: 31 · 2019: 16 · 2018: 19 · 2017: 12 · 2016: 12 · 2015: 17 · 2014: 21 · 2013: 16 · 2012: 16
- FWCI
- 2.22
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- iopscience.iop.org/page/copyright (vor, 29 d)
- iopscience.iop.org/info/page/text-and-data-mining (tdm, 29 d)
- Locations
- Journal of Statistical Mechanics Theory and Experiment (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- CERN Document Server (European Organization for Nuclear Research) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Social Sciences › Economics, Econometrics and Finance › Finance › Stochastic processes and financial applications
- Physical Sciences › Mathematics › Numerical Analysis › Numerical Solvers for Ordinary Differential Equations
- Keywords
- path integral, optimal control, linear quadratic control, stochastic control, Hamilton-Jacobi-Bellman equation, symmetry breaking, Pontryagin maximum principle, forward diffusion, Monte Carlo sampling, Laplace approximation, variational approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Path integrals and symmetry breaking for optimal control theory. https://app.cochat.ai/papers/10.1088/1742-5468/2005/11/p11011
- The paper
- Kappen, H. J. (2005). Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics Theory and Experiment. https://doi.org/10.1088/1742-5468/2005/11/p11011
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.