Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using reinforcement learning
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
To find the path that minimizes the time to navigate between two given points in a fluid flow is known as Zermelo’s problem. Here, we investigate it by using a Reinforcement Learning (RL) approach for the case of a vessel that has a slip velocity with fixed intensity, Vs, but variable direction and navigating in a 2D turbulent sea. We show that an Actor-Critic RL algorithm is able to find quasioptimal solutions for both time-independent and chaotically evolving flow configurations. For the frozen case, we also compared the results with strategies obtained analytically from continuous Optimal Navigation (ON) protocols. We show that for our application, ON solutions are unstable for the typical duration of the navigation process and are, therefore, not useful in practice. On the other hand, RL solutions are much more robust with respect to small changes in the initial conditions and to external noise, even when Vs is much smaller than the maximum flow velocity. Furthermore, we show how the RL approach is able to take advantage of the flow properties in order to reach the target, especially when the steering speed is small.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“The learning rates αt, α ′ t, follow the Adam algorithm [51] to improve the convergence performance over standard stochastic gradient descent.”
MethodHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 2×
“[54] V.”
MethodA Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
Machine Learning for Fluid Mechanics
Brunton, S. L., Noack, B. R., & Koumoutsakos, P. (2020) · Annual Review of Fluid Mechanics · doi:10.1146/annurev-fluid-010719-060214 · Cited by 3,063 · cited 2×
“Similarly, in the recent years, artificial intelligence techniques are establishing themselves as new data driven models for fluid mechanics in general [41–46].”
BackgroundActive Particles in Complex and Crowded Environments
Bechinger, C., Di Leonardo, R., Löwen, H., Reichhardt, C., Volpe, G., & Volpe, G. (2016) · Reviews of Modern Physics · doi:10.1103/revmodphys.88.045006 · Cited by 2,961 · cited 2×
“Path planning for small autonomous marine vehicles [1, 2] such as wave and current gliders [3, 4], active drifters [5, 6], buoyant underwater explorers, and small swimming drones is key for many geo-physical [7] and engineering [8–11] applications.”
BackgroundOtt, E. (2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511803260 · Cited by 2,298
Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,744
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 2×
“[55] Y.”
MethodThe dynamics of enstrophy transfer in two-dimensional hydrodynamics
Weiss, J. (1991) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(91)90088-q · Cited by 1,115
Research works printed but not linked: 7
- 1975, Applied Optimal Control: Optimization, Estimation and Control
- 2010, Optimal Control Theory with Aerospace Applications
- 2018, Reinforcement Learning: An Introduction
- 1996, Linear System Theory
- 2018, Mathematical Theory of Optimal Processes
- 2009, Chaos: From Simple Models to Complex Systems
- 2005, Dynamical Systems Approach to Turbulence
Cited by
Items 101–144 of 144
A mechanistic approach to plankton migration
Monthiller, R. (2022) · theses.fr (ABES) · doi:10.70675/b9095058z8af9z4252z8dd3za4fc13b532de · Cited by 3
A Review on Bio-inspired Fluid Mechanics via Deep Reinforcement Learning
Wang, J., Jiang, Z., Yang, Y. M., & Hu, W. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-1549-1_23 · Cited by 2
Programmable Control of Ultrasound Swarmbots through Reinforcement LearningPreprint
Schrage, M., Medany, M., & Ahmed, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15393 · Cited by 2
Reducing the Drag of a Bluff Body by Deep Reinforcement LearningPreprint
Ballini, E., Chiappa, A. S., & Micheletti, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03647 · Cited by 2
“Some particular applications are: PDE control [7, 16], flow control [46, 1, 15], control of the movement of objects immersed in a fluid, such as fish-like swimmers [35], microswimmers [34], Zermelo problem [2], drones e gliders [4, 44], as well as more general applications as shape optimization [57], and the creation…”
BackgroundIntermittent turbulent gusts lift eaglesPreprint
Gupta, D. P. S., Brandes, D. W., Lanzone, M., Miller, T., & Bewley, G. P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.00231 · Cited by 2
Can flocking aid the path planning of microswimmers in turbulent flows?
Gupta, A., Alageshan, J. K., Kiran, K. V., & Pandit, R. A. (2025) · Physics of Fluids · doi:10.1063/5.0254816 · Cited by 1
“Similar studies have been carried out for the gravitaxis [10] of mi-croswimmers and their path planning via actor-critic re-inforcement learning in two-dimensional (2D) turbulent flows [11].”
Adaptive shape control for microswimmer navigation in turbulence
Qiu, J., Piro, L., Biferale, L., Cencini, M., Mehlig, B., & Gustavsson, K. (2026) · Journal of Fluid Mechanics · doi:10.1017/jfm.2026.11923 · Cited by 1
Self-citationYoung, R., Akins, F. H., & Merrifield, S. T. (2026) · IEEE Journal of Oceanic Engineering · doi:10.1109/joe.2026.3692006 · Cited by 1
“In certain environments, the chaotic nature of optimal trajectories derived from ODE-based methods [24] increase the difficulty of selecting a suitable initial condition, particularly in higher dimensions.”
InfluentialOptimal navigation in two-dimensional flows: Control theory and reinforcement learning
Parfenyev, V. (2026) · Physical review. E · doi:10.1103/8gbr-vp2r · Cited by 1
Learning to traverse convective flows at moderate to high Rayleigh numbers
Xu, A., Wu, H.-L., Xu, B.-R., & Xi, H.-D. (2026) · Journal of Fluid Mechanics · doi:10.1017/jfm.2026.11802 · Cited by 1
Multiagent rendezvous in fluid flows via reinforcement learning
Li, B., Qiu, J., & Zhao, L. (2026) · Physical review. E · doi:10.1103/bdjn-fvyd · Cited by 1
Xu, A., Wu, H.-L., & Xi, H.-D. (2023) · IUTAM bookseries · doi:10.1007/978-3-031-47258-9_20 · Cited by 1
Yajima, T., & Tazawa, Y. (2024) · Journal of Optimization Theory and Applications · doi:10.1007/s10957-023-02374-2 · Cited by 1
Reinforcement Learning for Fluid Mechanics: an overview on Fundamentals from a Control PerspectivePreprint
Semeraro, O. (2024) · HAL · Cited by 1
Transport barriers for microswimmers in unsteady flowPreprint
Storm, L., Qiu, J., Gustavsson, K., & Mehlig, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.16430 · Cited by 1
“Recent work addresses this for tasks such as directional navigation [35–37], point-to-point navigation [38–41], clustering [42], and targeting specific flow regions [43], relying on heuristic signals with minimal prior information about the underlying dynamics.”
Self-citationBio-Inspired Sensing and Control of Micro Aerial Vehicles (MAVs)
Yang, H. (2023) · eCommons (Cornell University) · doi:10.7298/4n4e-9708 · Cited by 1
Navigation of micro-swimmers in steady flow: the importance of symmetries
Mehlig, B. (2022) · EGU General Assembly Conference Abstracts · doi:10.5194/egusphere-egu22-7811
Active gyrotactic stability of microswimmers using hydromechanical signals
Gustavsson, K., Qiu, J., Mousavi, N., & Zhao, L. (2022) · EGU General Assembly Conference Abstracts · doi:10.5194/egusphere-egu22-4812
Self-citationEfficiency of Navigation Strategies for Active Particles in Rugged Landscapes
Piro, L. (2024) · Springer theses · doi:10.1007/978-3-031-52577-3_3
Optimal Navigation Strategies for Active Particles on Curved Surfaces
Piro, L. (2024) · Springer theses · doi:10.1007/978-3-031-52577-3_2
Piro, L. (2024) · Springer theses · doi:10.1007/978-3-031-52577-3_1
Optimal Navigation in Complex and Noisy Environments
Piro, L. (2024) · Springer theses · doi:10.1007/978-3-031-52577-3_4
Masoudi, M. H., & Naji, A. (2025) · The Journal of Chemical Physics · doi:10.1063/5.0269792
Rossa, M. D., Freddi, L., & Pinatto, M. (2026) · Applied Mathematics & Optimization · doi:10.1007/s00245-026-10541-w
Whitepaper: Optimal Control from a Fluid Dynamics Perspective
(NNSA), U. N. N. S. A., Pratt, J., USNRC, Schneider, M. D., & Perloff, A. (2025) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/2998820
Optimal Control from a Fluid Dynamics PerspectivePreprint
Pratt, J., Schneider, M. D., & Perloff, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.17929
“This collection of works [Biferale et al., 2019, Buzzicotti et al., 2020, Calascibetta et al., 2023a,b] demonstrates two things: (1) that reinforcement learning based on the actor-critic algorithm can be used to solve this type of problem, i.e. the learning process produces robust and stable…”
Khalasi: Energy-Efficient Navigation for Surface Vehicles in Vortical Flow FieldsPreprint
Gadhvi, R., & Manjanna, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.06912
“Biferale et al. [9] extended RL to Zermelo’s navigation problem, showing robustness in turbulent regimes but relying on long-range flow data, which limits applicability in scenarios with restricted sensing.”
Global manoeuvring of confined active particle swarms via emergent threshold-based control (a)
Calascibetta, C., Giraldi, L., & Bec, J. (2026) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/ae51bc
Real Time Local Wind Inference for Robust Autonomous NavigationPreprint
Folk, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.00343
On the coupling of deep reinforcement learning and computational fluid dynamics
Ghraieb, H. (2022) · theses.fr (ABES) · doi:10.70675/df14f2d0z29e3z45a3z99a4z5c8561c0f6fb
A critical assessment of reinforcement learning methods for bio-inspired navigation
Mecanna, S. (2024) · theses.fr (ABES) · doi:10.70675/72f3a232z0ecfz4700z900az04c4c011ced5
Zhao, C., Zhao, D., Bian, X., & Li, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.18881
“…solely on locally sensed concentration and flow velocity information, the agent develops a hybrid strategy that combines the well-known cast-and-surge behavior [36, 38] with a tendency to move along the cross-flow direction, which is commonly observed for navigation in unsteady flows [39, 40].”
Berman, J., & Gal, O. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.25025
“Agents update their positions via explicit time integration of the acceleration.”
Liang, Z., Wang, Y., Zhang, X., & Tang, M. (2026) · Ocean Engineering · doi:10.1016/j.oceaneng.2026.126545
Autonomous Aerial Delivery of Flapping-Wing Flying Robots
Liu, C., Zhang, X., Meng, T., Wu, X., & He, X. (2026) · Youth Academic Annual Conference of Chinese Association of Automation (YAC) · doi:10.1109/yac71005.2026.11615326
“Aerial delivery is increasingly important in disaster relief, military logistics, and environmental monitoring, where pay-loads must be delivered accurately under complex conditions [1]–[3].”
Guided navigation in turbulencePreprint
Qiu, J., & Gustavsson, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30675
“3 and 4].”
Self-citationAnonymous, & Yang, Y. (2026) · Physical Review Fluids · doi:10.1103/g3qy-d7q7
González, J. A. (2026) · Foundations · doi:10.3390/foundations6040039
Learning minimum-time navigation policies in two-dimensional flows with a genetic algorithmPreprint
Parfenyev, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.12177
“Compared to tabular RL methods 24,25,36 , no distance-based potential is required in the reward.”
InfluentialDeep reinforcement learning-based UAV navigation in 3D urban boundary layers
Yan, L., & Liu, C. (2026) · Sustainable Cities and Society · doi:10.1016/j.scs.2026.107981
Meng, L., Yao, Q., & Zhang, Q. (2026) · WRC Symposium on Advanced Robotics and Automation (WRC SARA) · doi:10.1109/wrcsara71458.2026.11732543
“This problem is closely related to the Zermelo navigation problem [5], where a moving agent must select appropriate motion directions in a flow field to reach a goal efficiently.”
Zhu, Y., Pang, J.-H., & Tian, F.-B. (2022) · doi:10.3389/fphy.2022.870273.s009
Metadata
- Authors
- Biferale, L., Bonaccorso, F., Buzzicotti, M., Clark Di Leoni, P., & Gustavsson, K.
- Affiliations
- University of Rome Tor Vergata (IT)
- Department of Physics, INFN University of Rome Tor vergata 1 , via della Ricerca Scientifica 1, 00133 Rome, Italy
- University of Gothenburg (SE)
- Venue
- Chaos: An Interdisciplinary Journal of Nonlinear Science
- Publisher
- AIP Publishing
- Published
- 2019-10-01
- Volume
- 29
- Issue
- 10
- Pages
- 103138-103138
- ISSN
- 1054-1500 (ISSN-L)
- 1089-7682
- OpenAlex
- W2963604378
- Cited by
- 121
- References
- 66
- Citations by year
- 2026: 13 · 2025: 21 · 2024: 20 · 2023: 22 · 2022: 20 · 2021: 15 · 2020: 9
- FWCI
- 5.02
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Chaos An Interdisciplinary Journal of Nonlinear Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Cineca Institutional Research Information System (Tor Vergata University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- European Union Programme (339032)
- Knut och Alice Wallenbergs Stiftelse (KAW 2014.0048)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Ocean Engineering › Maritime Navigation and Safety
- Keywords
- turbulent flow, reinforcement learning, actor-critic algorithms, slip velocity, two-dimensional turbulence, noise robustness, chaotic flow, point-to-point navigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.