Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using reinforcement learning
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
To find the path that minimizes the time to navigate between two given points in a fluid flow is known as Zermelo’s problem. Here, we investigate it by using a Reinforcement Learning (RL) approach for the case of a vessel that has a slip velocity with fixed intensity, Vs, but variable direction and navigating in a 2D turbulent sea. We show that an Actor-Critic RL algorithm is able to find quasioptimal solutions for both time-independent and chaotically evolving flow configurations. For the frozen case, we also compared the results with strategies obtained analytically from continuous Optimal Navigation (ON) protocols. We show that for our application, ON solutions are unstable for the typical duration of the navigation process and are, therefore, not useful in practice. On the other hand, RL solutions are much more robust with respect to small changes in the initial conditions and to external noise, even when Vs is much smaller than the maximum flow velocity. Furthermore, we show how the RL approach is able to take advantage of the flow properties in order to reach the target, especially when the steering speed is small.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“The learning rates αt, α ′ t, follow the Adam algorithm [51] to improve the convergence performance over standard stochastic gradient descent.”
MethodHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 2×
“[54] V.”
MethodA Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
Machine Learning for Fluid Mechanics
Brunton, S. L., Noack, B. R., & Koumoutsakos, P. (2020) · Annual Review of Fluid Mechanics · doi:10.1146/annurev-fluid-010719-060214 · Cited by 3,063 · cited 2×
“Similarly, in the recent years, artificial intelligence techniques are establishing themselves as new data driven models for fluid mechanics in general [41–46].”
BackgroundActive Particles in Complex and Crowded Environments
Bechinger, C., Di Leonardo, R., Löwen, H., Reichhardt, C., Volpe, G., & Volpe, G. (2016) · Reviews of Modern Physics · doi:10.1103/revmodphys.88.045006 · Cited by 2,961 · cited 2×
“Path planning for small autonomous marine vehicles [1, 2] such as wave and current gliders [3, 4], active drifters [5, 6], buoyant underwater explorers, and small swimming drones is key for many geo-physical [7] and engineering [8–11] applications.”
BackgroundOtt, E. (2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511803260 · Cited by 2,298
Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,744
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 2×
“[55] Y.”
MethodThe dynamics of enstrophy transfer in two-dimensional hydrodynamics
Weiss, J. (1991) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(91)90088-q · Cited by 1,115
Research works printed but not linked: 7
- 1975, Applied Optimal Control: Optimization, Estimation and Control
- 2010, Optimal Control Theory with Aerospace Applications
- 2018, Reinforcement Learning: An Introduction
- 1996, Linear System Theory
- 2018, Mathematical Theory of Optimal Processes
- 2009, Chaos: From Simple Models to Complex Systems
- 2005, Dynamical Systems Approach to Turbulence
Cited by
Machine learning for active matter
Cichos, F., Gustavsson, K., Mehlig, B., & Volpe, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0146-9 · Cited by 315
Self-citationTang, Y. S., Kurths, J., Lin, W. J., Ott, E., & Kocarev, L. (2020) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0016505 · Cited by 199
Automating turbulence modelling by multi-agent reinforcement learning
Novati, G., de Laroussilhe, H. L., & Koumoutsakos, P. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00272-0 · Cited by 182
Scientific multi-agent reinforcement learning for wall-models of turbulent flows
Bae, H. J., & Koumoutsakos, P. (2022) · Nature Communications · doi:10.1038/s41467-022-28957-7 · Cited by 171
Rabault, J., Ren, F., Zhang, W., Tang, H., & Xu, H. (2020) · Journal of Hydrodynamics · doi:10.1007/s42241-020-0028-y · Cited by 171
Deep reinforcement learning for turbulent drag reduction in channel flows
Guastoni, L., Rabault, J., Schlatter, P., Azizpour, H., & Vinuesa, R. (2023) · The European Physical Journal E · doi:10.1140/epje/s10189-023-00285-8 · Cited by 164
Machine learning for micro- and nanorobots
Yang, L., Jiang, J., Ji, F., Li, Y., Yung, K. L., Ferreira, A., & Zhang, L. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00859-x · Cited by 125
Controlling Rayleigh–Bénard convection via reinforcement learning
Beintema, G., Corbetta, A., Biferale, L., & Toschi, F. (2020) · Journal of Turbulence · doi:10.1080/14685248.2020.1797059 · Cited by 118
Self-citationLearning efficient navigation in vortical flow fields
Gunnarson, P., Mandralis, I., Novati, G., Koumoutsakos, P., & Dabiri, J. O. (2021) · Nature Communications · doi:10.1038/s41467-021-27015-y · Cited by 108
Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows
Ghraieb, H., Viquerat, J., Larcher, A., Méliga, P., & Hachem, E. (2021) · Physical Review Fluids · doi:10.1103/physrevfluids.6.053902 · Cited by 91
Machine learning strategies for path-planning microswimmers in turbulent flows
Alageshan, J. K., Verma, A. K., Bec, J., & Pandit, R. (2020) · Physical Review E · doi:10.1103/physreve.101.043110 · Cited by 88
SINDy-RL for interpretable and efficient model-based reinforcement learning
Zolman, N., Lagemann, C., Fasel, U., Kutz, J. N., & Brunton, S. L. (2025) · Nature Communications · doi:10.1038/s41467-025-65738-4 · Cited by 71
Deep reinforcement learning for the control of conjugate heat transfer
Hachem, E., Ghraieb, H., Viquerat, J., Larcher, A., & Méliga, P. (2021) · Journal of Computational Physics · doi:10.1016/j.jcp.2021.110317 · Cited by 70
Gait switching and targeted navigation of microswimmers via deep reinforcement learning
Zou, Z., Liu, Y., Young, Y., Pak, O. S., & Tsang, A. C. H. (2022) · Communications Physics · doi:10.1038/s42005-022-00935-x · Cited by 64
Jarosiński, M., Beekman, F., Bada, G., & Cloetingh, S. A. P. L. (2006) · Geophysical Journal International · doi:10.1111/j.1365-246x.2006.02979.x · Cited by 64
Hydrodynamics can determine the optimal route for microswimmer navigation
Daddi‐Moussa‐Ider, A., Löwen, H., & Liebchen, B. (2021) · Communications Physics · doi:10.1038/s42005-021-00522-6 · Cited by 61
Micro/Nano Motor Navigation and Localization via Deep Reinforcement Learning
Yang, Y., Bevan, M. A., & Li, B. (2020) · Advanced Theory and Simulations · doi:10.1002/adts.202000034 · Cited by 58
Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number
Borra, F., Biferale, L., Cencini, M., & Celani, A. (2022) · Physical Review Fluids · doi:10.1103/physrevfluids.7.023103 · Cited by 54
Self-citationReinforcement learning of optimal active particle navigation
Nasiri, M., & Liebchen, B. (2022) · New Journal of Physics · doi:10.1088/1367-2630/ac8013 · Cited by 53
Cheng, H., Li, H., Dai, Q., & Yang, J. (2023) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2023.113809 · Cited by 52
Navigation of micro-swimmers in steady flow: the importance of symmetries
Qiu, J., Mousavi, N., Gustavsson, K., Xu, C., Mehlig, B., & Zhao, L. (2021) · Journal of Fluid Mechanics · doi:10.1017/jfm.2021.978 · Cited by 52
“…between navigation using local signals in the frame of reference of the swimmer and signals in the laboratory frame (Colabrese et al. 2017, 2018; Gustavsson et al. 2017; Biferale et al. 2019; Schneider & Stark 2019; Alageshan et al. 2020; Gunnarson et al. 2021; Muiños-Landin et al. 2021).”
BackgroundSelf-citationOptimal active particle navigation meets machine learning (a)
Nasiri, M., Löwen, H., & Liebchen, B. (2023) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/acc270 · Cited by 42
Data reconstruction for complex flows using AI: Recent progress, obstacles, and perspectives
Buzzicotti, M. (2023) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/acc88c · Cited by 40
Self-citationEbers, M. R., Williams, J. P., Steele, K. M., & Kutz, J. N. (2024) · IEEE Access · doi:10.1109/access.2024.3423679 · Cited by 32
“Path planning is a long-standing challenge in the field of engineering and robotics related to navigation and estimation in dynamical environments [24], [25], [26], [27], [28], [29].”
BackgroundActive gyrotactic stability of microswimmers using hydromechanical signals
Qiu, J., Mousavi, N., Zhao, L., & Gustavsson, K. (2022) · Physical Review Fluids · doi:10.1103/physrevfluids.7.014311 · Cited by 32
Self-citationOptimal navigation of microswimmers in complex and noisy environments
Piro, L., Mahault, B., & Golestanian, R. (2022) · New Journal of Physics · doi:10.1088/1367-2630/ac9079 · Cited by 29
Optimal navigation strategies for microswimmers on curved manifolds
Piro, L., Tang, E., & Golestanian, R. (2021) · Physical Review Research · doi:10.1103/physrevresearch.3.023125 · Cited by 28
Mechanical rotation at low Reynolds number via reinforcement learning
Liu, Y., Zou, Z., Tsang, A. C. H., Pak, O. S., & Young, Y. (2021) · Physics of Fluids · doi:10.1063/5.0053563 · Cited by 28
Karnakov, P., Litvinov, S., & Koumoutsakos, P. (2023) · The European Physical Journal E · doi:10.1140/epje/s10189-023-00313-7 · Cited by 28
“In the context of optimal control, the flow can be sampled from the forward problem to obtain a policy via reinforcement learning [14].”
BackgroundActive particles using reinforcement learning to navigate in complex motility landscapes
Monderkamp, P. A., Schwarzendahl, F. J., Klatt, M. A., & Löwen, H. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/aca7b0 · Cited by 27
Buzzicotti, M., Biferale, L., Bonaccorso, F., Leoni, P. C. D., & Gustavsson, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77091-4_14 · Cited by 26
Self-citationOptimal tracking strategies in a turbulent flow
Calascibetta, C., Biferale, L., Borra, F., Celani, A., & Cencini, M. (2023) · Communications Physics · doi:10.1038/s42005-023-01366-y · Cited by 26
Self-citationReinforcement learning for patient-specific optimal stenting of intracranial aneurysms
Hachem, E., Méliga, P., Goetz, A., Rico, P. J., Viquerat, J., Larcher, A., Valette, R., Fava-Sanches, A., Lannelongue, V., Ghraieb, H., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-34007-z · Cited by 24
Finite-horizon, energy-efficient trajectories in unsteady flows
Krishna, K., Song, Z., & Brunton, S. L. (2022) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2021.0255 · Cited by 24
Sensing flow gradients is necessary for learning autonomous underwater navigation
Jiao, Y., Hang, H., Merel, J., & Kanso, E. (2025) · Nature Communications · doi:10.1038/s41467-025-58125-6 · Cited by 23
Koopman-Assisted Reinforcement LearningPreprint
Rozwood, P., Mehrez, E., Paehler, L., Sun, W., & Brunton, S. L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.02290 · Cited by 23
“Mathematically, given a function from the state space to the reals g : X → R , the Koopman operator K for deterministic (time-homogenous) autonomous systems is defined as: K g ( x ) := g ( F ( x )) = g ( x (cid:48) ) , (1) where F is the single-step flow map or law of motion.”
Navigation in a simplified urban flow through deep reinforcement learning
Tonti, F., Rabault, J., & Vinuesa, R. (2025) · Journal of Computational Physics · doi:10.1016/j.jcp.2025.114194 · Cited by 22
Zhu, Y. F., Pang, J., & Tian, F.-B. (2022) · Fluids · doi:10.3390/fluids7010041 · Cited by 22
Cavallazzi, G. M., Guastoni, L., Vinuesa, R., & Pinelli, A. (2024) · Flow Turbulence and Combustion · doi:10.1007/s10494-024-00609-4 · Cited by 20
“For instance, some studies represent a fish swimming in turbulent flow or within a school of fish, where the reward function is designed to maximise the swimming efficiency (Biferale et al. 2019; Verma et al. 2018).”
Taming Lagrangian chaos with multi-objective reinforcement learning
Calascibetta, C., Biferale, L., Borra, F., Celani, A., & Cencini, M. (2023) · The European Physical Journal E · doi:10.1140/epje/s10189-023-00271-0 · Cited by 19
“In this direction, a few attempts to control single Lagrangian instrumented particles via Reinforcement Learning (RL) algorithms have been proposed to solve the Zermelo’s optimal navigation problem of reaching a fixed target [12,13,14].”
MethodSelf-citationEnergetic cost of microswimmer navigation: The role of body shape
Piro, L., Vilfan, A., Golestanian, R., & Mahault, B. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.013274 · Cited by 19
Chemotactic navigation in robotic swimmers via reset-free hierarchical reinforcement learning
Xiong, T., Liu, Z., Wang, Y., Ong, C. J., & Zhu, L. (2025) · Nature Communications · doi:10.1038/s41467-025-60646-z · Cited by 19
Point-to-Point Navigation of a Fish-Like Swimmer in a Vortical Flow With Deep Reinforcement Learning
Zhu, Y. F., Pang, J., & Tian, F.-B. (2022) · Frontiers in Physics · doi:10.3389/fphy.2022.870273 · Cited by 19
Lai, P., Wang, R., Zhang, W., & Xu, H. (2021) · Physics of Fluids · doi:10.1063/5.0068454 · Cited by 18
Optimal navigation of a smart active particle: directional and distance sensing
Putzke, M., & Stark, H. (2023) · The European Physical Journal E · doi:10.1140/epje/s10189-023-00309-3 · Cited by 18
Efficiency of navigation strategies for active particles in rugged landscapes
Piro, L., Golestanian, R., & Mahault, B. (2022) · Frontiers in Physics · doi:10.3389/fphy.2022.1034267 · Cited by 17
Long-distance migration with minimal energy consumption in a thermal turbulent environment
Xu, A., Wu, H.-L., & Xi, H.-D. (2023) · Physical Review Fluids · doi:10.1103/physrevfluids.8.023502 · Cited by 17
Smart active particles learn and transcend bacterial foraging strategies
Nasiri, M., Loran, E., & Liebchen, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2317618121 · Cited by 16
Efficient navigation of a robotic fish swimming across the vortical flow field
Feng, H., Yuan, D., Miao, J.-L., You, J., Wang, Y., Zhu, Y. F., & Fan, D. (2024) · Journal of Hydrodynamics · doi:10.1007/s42241-025-0103-5 · Cited by 16
“Secondly, an idealized representation of the fishes and their actions was common [19, 36, 37].”
Optimal navigation and behavioural traits in oceanic migrations
Pinti, J., Celani, A., Thygesen, U. H., & Mariani, P. (2020) · Theoretical Ecology · doi:10.1007/s12080-020-00469-4 · Cited by 16
“relation to migration before (e.g. Hays et al. 2014a; McLaren et al. 2014; Biferale et al. 2019), but they did not consider time and energy constraints together and the different optimal strategies that may arise due to behavioural discrepancies between individuals.”
BackgroundTomasetto, M., Braghin, F., & Manzoni, A. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.118030 · Cited by 15
Flow currents support simple and versatile trail-tracking strategies
Hang, H., Jiao, Y., Merel, J., & Kanso, E. A. (2025) · Physical Review Research · doi:10.1103/qj9q-p7rg · Cited by 15
Towards Intelligent Active Particles
Löwen, H., & Liebchen, B. (2026) · Machine intelligence for materials science · doi:10.1007/978-3-032-04129-6_13 · Cited by 15
Autonomous Underwater Vehicle Path Planning Method of Soft Actor–Critic Based on Game Training
Wang, Z., Lu, H., Qin, H., & Sui, Y. (2022) · Journal of Marine Science and Engineering · doi:10.3390/jmse10122018 · Cited by 15
Physics-guided actor-critic reinforcement learning for swimming in turbulence
Koh, C., Pagnier, L., & Chertkov, M. (2025) · Physical Review Research · doi:10.1103/physrevresearch.7.013121 · Cited by 14
“A particularly in-triguing direction is the control of swarms of Lagrangian particles with contrasting objectives, such as maintaining group cohesion, avoiding close encounters, and minimizing energy expenditure for maneuvering.”
Zhang, R., Qin, X., Pan, M., Li, S., & Shen, H. (2025) · Journal of Marine Science and Engineering · doi:10.3390/jmse13030514 · Cited by 14
Optimizing collective behavior of communicating active particles with machine learning
Grauer, J., Schwarzendahl, F. J., Löwen, H., & Liebchen, B. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad1c33 · Cited by 14
Learning how to find targets in the micro-world: the case of intermittent active Brownian particles
Caraglio, M., Kaur, H., Fiderer, L. J., López-Incera, A., Briegel, H. J., Franosch, T., & Muñoz-Gil, G. (2024) · Soft Matter · doi:10.1039/d3sm01680c · Cited by 14
Dissipation-accuracy tradeoffs in autonomous control of smart active matter
Cocconi, L., Mahault, B., & Piro, L. (2024) · New Journal of Physics · doi:10.1088/1367-2630/ad9f9a · Cited by 13
Optimizing airborne wind energy with reinforcement learning
Orzan, N., Leone, C., Mazzolini, A., Oyero, J. O., & Celani, A. (2023) · The European Physical Journal E · doi:10.1140/epje/s10189-022-00259-2 · Cited by 13
Mobile Sensor Path Planning for Kalman Filter Spatiotemporal Estimation
Mei, J., Brunton, S. L., & Kutz, J. N. (2024) · Sensors · doi:10.3390/s24123727 · Cited by 13
Finite Time Lyapunov Exponent Analysis of Model Predictive Control and Reinforcement Learning
Krishna, K., Brunton, S. L., & Song, Z. (2023) · IEEE Access · doi:10.1109/access.2023.3326424 · Cited by 12
“Recent developments have made use of model predictive control (MPC) [37] and reinforcement learning (RL) algorithms [38], [39], [40], [41], [42] to find optimal paths in unsteady flow fields.”
BackgroundOptimal Navigation in Microfluidics via the Optimization of a Discrete Loss
Karnakov, P., Amoudruz, L., & Koumoutsakos, P. (2025) · Physical Review Letters · doi:10.1103/physrevlett.134.044001 · Cited by 12
“In comparison to RL, which currently represents the state-of-the-art for path planning and control problems in these applications [34, 35, 36, 6, 48, 49, 37], ODIL is more robust, in particular for high-dimensional state/action spaces while requiring between one to three orders of magnitude fewer…”
Amoudruz, L., Litvinov, S., & Koumoutsakos, P. (2025) · Physics of Fluids · doi:10.1063/5.0274623 · Cited by 12
“Similarly, several studies applied RL to guide microswimmers [17] through background flows [18, 19, 20, 21, 22] and perform independent control of multiple ABFs [23, 24] to reach a target or follow a prescribed path [25].”
Learning to swim efficiently in a nonuniform flow field
Sankaewtong, K., Molina, J. J., Turner, M. S., & Yamamoto, R. (2023) · Physical review. E · doi:10.1103/physreve.107.065102 · Cited by 12
Fish-inspired tracking of underwater turbulent plumes
Gunnarson, P., & Dabiri, J. O. (2024) · Bioinspiration & Biomimetics · doi:10.1088/1748-3190/ad7181 · Cited by 11
Efficient navigation in vortical flows based on reinforcement learning and flow field prediction
Zhang, Y., Zheng, S., Xu, C., & Cai, S. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.120937 · Cited by 11
A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows
Mecanna, S., Loisy, A., & Eloy, C. (2025) · The European Physical Journal E · doi:10.1140/epje/s10189-025-00522-2 · Cited by 11
“Similar 2D turbulent flows have been used in Refs [16, 31].”
Zhu, Y. F., & Pang, J. (2022) · Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science · doi:10.1177/09544062221079693 · Cited by 11
Short term vs. long term: Optimization of microswimmer navigation on different time horizons
Mousavi, N., Qiu, J., Zhao, L., Mehlig, B., & Gustavsson, K. (2025) · Physical Review Research · doi:10.1103/physrevresearch.7.013258 · Cited by 10
“(1c) and (1d) allow the swimmer to actively swim and steer [25, 27, 28, 35, 36].”
Self-citationChu, S., Feng, H., Ou, Y., Lin, M., & Li, D. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.123188 · Cited by 10
An Account on Links Between Finsler and Lorentz Geometries for Riemannian Geometers
Javaloyes, M. Á., Pendás-Recondo, E., & Sánchez, M. (2023) · RSME Springer series · doi:10.1007/978-3-031-39916-9_10 · Cited by 10
“Zermelo problem can also be solved using Optimal Control Theory (see the classical book [16] or [12, 14, 15, 74] for recent developments), but our interest will focus on more geometrical methods, namely, the use of Finsler Geometry to solve the problem.”
MethodEfficient survival strategy for zooplankton in turbulence
Mousavi, N., Qiu, J., Mehlig, B., Zhao, L., & Gustavsson, K. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.l022034 · Cited by 9
Self-citationLai, J., Ren, Z., Wu, Z., Tan, Q., & Xie, S. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.121505 · Cited by 9
Super-resolution reconstruction of turbulent flows from a single Lagrangian trajectory
Wu, H.-L., Xu, A., & Xi, H.-D. (2026) · Journal of Fluid Mechanics · doi:10.1017/jfm.2025.11033 · Cited by 9
Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics
Berger, S., Ramo, A. A., Guillet, V., Lahire, T., Martin, B., Jardin, T., Rachelson, E., & Bauerheim, M. (2024) · Data-Centric Engineering · doi:10.1017/dce.2023.28 · Cited by 9
Informational active matterPreprint
VanSaders, B. J., & Vitelli, V. (2023) · Research Square · doi:10.21203/rs.3.rs-2592455/v1 · Cited by 9
Capillary hypersurfaces, Heintze–Karcher’s inequality and Zermelo’s navigation
Wang, G., & Xia, C. (2024) · Calculus of Variations and Partial Differential Equations · doi:10.1007/s00526-024-02837-y · Cited by 8
“Here we just mention a few: [13] in stationary spacetimes, [14] for a model for traffic noise with magnetic flow, [4] for 2D turbulent flows, [9] in wildfire spread modelling, and [6, 28] in the quantum Zermelo problem [6, 28].”
Inferring turbulent environments via machine learning
Buzzicotti, M., & Bonaccorso, F. (2022) · The European Physical Journal E · doi:10.1140/epje/s10189-022-00258-3 · Cited by 8
Self-citationArtificial intelligence (AI) enhanced nanomotors and active matter
Cichos, F., Muíños-Landín, S., & Pradip, R. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-85796-3.00005-6 · Cited by 8
Artificial microswimmers get smart
Stark, H. (2021) · Science Robotics · doi:10.1126/scirobotics.abh1977 · Cited by 8
“The advantage of this approach compared with traditional optimization via variational calculus is that it guides the swimmer particle back to the optimal path once thermal or any other noise disturbs it ( 9 , 10 ).”
Towards Safe and Energy-Efficient Real-Time Motion Planning in Windy Urban Environments
Folk, S., Melton, J., Margolis, B. W. L., Yim, M. H., & Kumar, V. R. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127986 · Cited by 7
“More recently, there have been examples of energy-aware trajectory optimization [13], [14] and reinforcement learning (RL) [15]–[17] within complex flow environments.”
Migration of self-propelling agent in a turbulent environment with minimal energy consumption
Xu, A., Wu, H.-L., & Xi, H.-D. (2022) · Physics of Fluids · doi:10.1063/5.0082845 · Cited by 7
Adaptive active Brownian particles searching for targets of unknown positions
Kaur, H., Franosch, T., & Caraglio, M. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ace6f4 · Cited by 7
Havenstrøm, S. T., Sterud, C., Rasheed, A., & San, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.01022 · Cited by 7
“For path following, as in the previously mentioned fields, DRL approaches have proven to be robust and successful potential alternatives to traditional algorithms ([17, 51, 52]), particularly in complex environments [3, 9, 47].”
BackgroundGeneralized synchronization between two distinct chaotic systems through deep reinforcement learning
Cheng, H., Li, H., Liang, J., Dai, Q., & Yang, J. (2025) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2025.116727 · Cited by 6
Jiren, M., Yousif, M. Z., Song, J., & Lim, H.-C. (2025) · Physics of Fluids · doi:10.1063/5.0297900 · Cited by 6
Physical embodiment enables information processing beyond explicit flow sensing in active matter
Paul, D., Milošević, N., Scherf, N., & Cichos, F. (2026) · Science Advances · doi:10.1126/sciadv.aec0783 · Cited by 6
“While simulations have shown RL agents can navigate complex flows [27–29] with recent experiments demonstrating learned navigation in controlled settings [21, 25, 30], no experimental system has validated whether microswimmers can infer and respond to hidden environmental perturbations using only…”
Mei, J., Kutz, J. N., & Brunton, S. L. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10383428 · Cited by 6
A Study of First-Passage Time Minimization via Q-Learning in Heated Gridworlds
Larchenko, M., Osinenko, P., Yaremenko, G., & Palyulin, V. V. (2021) · IEEE Access · doi:10.1109/access.2021.3129709 · Cited by 5
“RL was also applied for the navigation of microswimmers in such highly stochastic environments as complex and turbulent flows in [15–19].”
MethodA Fast-Tracking-Particle-Inspired Flow-Aided Control Approach for Air Vehicles in Turbulent Flow
Yang, H., Bewley, G. P., & Ferrari, S. (2022) · Biomimetics · doi:10.3390/biomimetics7040192 · Cited by 5
Drag-reduction strategies in wall-bounded turbulent flows using deep reinforcement learningPreprint
Guastoni, L., Rabault, J., Azizpour, H., & Vinuesa, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.02943 · Cited by 5
Time-dependent Zermelo navigation with tackingPreprint
Markvorsen, S., Pendás-Recondo, E., & Rygaard, F. M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.07274 · Cited by 5
“More recently, it has also been studied and solved using tools from optimal control theory (see e.g. [4 , 5, 36, 37, 50]).”
Intent matters: how flow and forms of information impact collective navigation
Hodgson, T. M., Johnston, S. T., Ottobre, M., & Painter, K. J. (2023) · Journal of The Royal Society Interface · doi:10.1098/rsif.2023.0356 · Cited by 4
Optimal navigation in active matter
Piro, L. (2023) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-9986 · Cited by 4
Run-and-tumble particles learning chemotaxis
Tovazzi, N., Muñoz-Gil, G., & Caraglio, M. (2025) · Soft Matter · doi:10.1039/d5sm01040c · Cited by 3
Jafarimoghaddam, A., & Soler, M. (2024) · Applied Mathematical Modelling · doi:10.1016/j.apm.2024.07.004 · Cited by 3
Applying Reinforcement Learning to Navigation In Partially Observable Flows
Mecanna, S., Loisy, A., & Eloy, C. (2024) · HAL · Cited by 3
Time Optimal Trajectory Planning of Unmanned Surface Vessels: A Deep Learning-based Control Approach
Lai, J., Ren, Z., & Wu, Z. (2021) · China Automation Congress (CAC) · doi:10.1109/cac53003.2021.9727826 · Cited by 3
“There are numerous methods available in the literature that can be used to tackle this constrained OPC, such as discretization method [7], control parameterizations method [8], reinforcement learning [9], [10].”
MethodReinforcement learning reveals fundamental limits on the mixing of active particles
Schildknecht, D., Popova, A. N., Stellwagen, J., & Thomson, M. W. (2021) · Soft Matter · doi:10.1039/d1sm01400e · Cited by 3
Metadata
- Authors
- Biferale, L., Bonaccorso, F., Buzzicotti, M., Clark Di Leoni, P., & Gustavsson, K.
- Affiliations
- University of Rome Tor Vergata (IT)
- Department of Physics, INFN University of Rome Tor vergata 1 , via della Ricerca Scientifica 1, 00133 Rome, Italy
- University of Gothenburg (SE)
- Venue
- Chaos: An Interdisciplinary Journal of Nonlinear Science
- Publisher
- AIP Publishing
- Published
- 2019-10-01
- Volume
- 29
- Issue
- 10
- Pages
- 103138-103138
- ISSN
- 1054-1500 (ISSN-L)
- 1089-7682
- OpenAlex
- W2963604378
- Cited by
- 121
- References
- 66
- Citations by year
- 2026: 13 · 2025: 21 · 2024: 20 · 2023: 22 · 2022: 20 · 2021: 15 · 2020: 9
- FWCI
- 5.02
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Chaos An Interdisciplinary Journal of Nonlinear Science (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Cineca Institutional Research Information System (Tor Vergata University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- European Union Programme (339032)
- Knut och Alice Wallenbergs Stiftelse (KAW 2014.0048)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Ocean Engineering › Maritime Navigation and Safety
- Keywords
- turbulent flow, reinforcement learning, actor-critic algorithms, slip velocity, two-dimensional turbulence, noise robustness, chaotic flow, point-to-point navigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.