Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Learning efficient navigation in vortical flow fields

Open access

Abstract

Efficient point-to-point navigation in the presence of a background flow field is important for robotic applications such as ocean surveying. In such applications, robots may only have knowledge of their immediate surroundings or be faced with time-varying currents, which limits the use of optimal control techniques. Here, we apply a recently introduced Reinforcement Learning algorithm to discover time-efficient navigation policies to steer a fixed-speed swimmer through unsteady two-dimensional flow fields. The algorithm entails inputting environmental cues into a deep neural network that determines the swimmer’s actions, and deploying Remember and Forget Experience Replay. We find that the resulting swimmers successfully exploit the background flow to reach the target, but that this success depends on the sensed environmental cue. Surprisingly, a velocity sensing approach significantly outperformed a bio-mimetic vorticity sensing approach, and achieved a near 100% success rate in reaching the target locations while approaching the time-efficiency of optimal navigation trajectories.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-021-27015-y

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Test navigation problem of navigating through unsteady cylinder flow. Swimmers are initialized randomly inside the red disk and are assigned a random target location inside the green disk. These regions of start and target points are 4D in diameter, and are located 5D downstream and centered 2.05D above and below the cylinder. Additionally, each swimmer is initialized at a random time step in the vortex shedding cycle. An episode is successful when a swimmer reaches within a radius of D/6 around the target location.
Fig. 1 Test navigation problem of navigating through unsteady cylinder flow. Swimmers are initialized randomly inside the red disk and are assigned a random target location inside the green disk. These regions of start and target points are 4D in diameter, and are located 5D downstream and centered 2.05D above and below the cylinder. Additionally, each swimmer is initialized at a random time step in the vortex shedding cycle. An episode is successful when a swimmer reaches within a radius of D/6 around the target location.CC BY · PubMed Central
Fig. 2 Evolution of the cumulative reward during training for the three RL swimmers. The cumulative rewards for each episode are plotted as points, and a moving average with a window of 201 episodes is plotted with a solid line. Because the swimmer gains a bonus of 200 for reaching the target, successful episodes are clustered around a reward of 200 while unsuccessful episodes are clustered below zero.
Fig. 2 Evolution of the cumulative reward during training for the three RL swimmers. The cumulative rewards for each episode are plotted as points, and a moving average with a window of 201 episodes is plotted with a solid line. Because the swimmer gains a bonus of 200 for reaching the target, successful episodes are clustered around a reward of 200 while unsuccessful episodes are clustered below zero.CC BY · PubMed Central
Fig. 3 Example trajectory of the velocity RL swimmer. a Trajectory plotted in a cylinder-fixed frame, showing the swimmer successfully navigate from its starting location to the target. b Segment of this trajectory plotted in a wake-stationary frame of reference on top of the background flow field, which highlights the swimmer exploiting low-velocity regions in the cylinder wake to swim upstream. The swimming direction is plotted at each time step along the trajectory, revealing that this RL swimmer adjusts it swimming direction in response to the changing background flow, enabling time-efficient navigation.
Fig. 3 Example trajectory of the velocity RL swimmer. a Trajectory plotted in a cylinder-fixed frame, showing the swimmer successfully navigate from its starting location to the target. b Segment of this trajectory plotted in a wake-stationary frame of reference on top of the background flow field, which highlights the swimmer exploiting low-velocity regions in the cylinder wake to swim upstream. The swimming direction is plotted at each time step along the trajectory, revealing that this RL swimmer adjusts it swimming direction in response to the changing background flow, enabling time-efficient navigation.CC BY · PubMed Central
Fig. 4 Average success rate with 30 example trajectories for each swimmer type. Successful attempts to reach the target are green, while unsuccessful attempts are red. a Naive policy of swimming towards the target is rarely successful. b The flow-blind RL swimmer navigates more effectively than the naive swimmer. c The vorticity RL swimmer is more successful than the flow-blind swimmer, showing that sensing the local flow can improve RL-based navigation. d Surprisingly, the velocity RL swimmer nearly always reaches the target using only the local flow velocity. The stated success rates are averaged over 12,500 episodes and are shown with one standard deviation arising from the five times each swimmer was trained.
Fig. 4 Average success rate with 30 example trajectories for each swimmer type. Successful attempts to reach the target are green, while unsuccessful attempts are red. a Naive policy of swimming towards the target is rarely successful. b The flow-blind RL swimmer navigates more effectively than the naive swimmer. c The vorticity RL swimmer is more successful than the flow-blind swimmer, showing that sensing the local flow can improve RL-based navigation. d Surprisingly, the velocity RL swimmer nearly always reaches the target using only the local flow velocity. The stated success rates are averaged over 12,500 episodes and are shown with one standard deviation arising from the five times each swimmer was trained.CC BY · PubMed Central
Fig. 5 Swimming direction policy plotted across the domain for a fixed target (green circle) at a given time instant. a The naive swimmer swims towards the target. b The red outline highlights how the flow-blind swimmer navigates irrespective of the background flow, while the vorticity swimmer c adjusts its swimming direction modestly. d The velocity swimmer appears even more sensitive to the unsteady background flow.
Fig. 5 Swimming direction policy plotted across the domain for a fixed target (green circle) at a given time instant. a The naive swimmer swims towards the target. b The red outline highlights how the flow-blind swimmer navigates irrespective of the background flow, while the vorticity swimmer c adjusts its swimming direction modestly. d The velocity swimmer appears even more sensitive to the unsteady background flow.CC BY · PubMed Central
Fig. 6 Comparison between time-optimal and RL trajectories. Time-optimal trajectories are shown in red and RL trajectories are shown in black. The RL swimmer used the state s = {Δx, Δy, u, v}. Time to reach the target Tf is made non-dimensional using the timescale D/U∞.
Fig. 6 Comparison between time-optimal and RL trajectories. Time-optimal trajectories are shown in red and RL trajectories are shown in black. The RL swimmer used the state s = {Δx, Δy, u, v}. Time to reach the target Tf is made non-dimensional using the timescale D/U∞.CC BY · PubMed Central
Fig. 7 RL navigation in the double gyre flow field. a Navigation problem setup. The start and target regions are L/2 in diameter and located at (3L/2, L/2) and (L/2, L/2), respectively. b A naive policy achieves 40.9% success rate on average. c The velocity RL swimmer trained on the cylinder wake navigates the double gyre flow poorly, indicating its navigation policy did not generalize. d After receiving training for the double gyre flow, the velocity RL swimmer is able to adapt and navigate more effectively than either swimmer. As with the cylinder flow, successful attempts to reach the target are green, while unsuccessful attempts are red. An episode is successful when a swimmer reaches within a radius of L/50 around the target location. The stated success rates are averaged over 12,500 episodes and are shown with one standard deviation arising from the five times each swimmer was trained.
Fig. 7 RL navigation in the double gyre flow field. a Navigation problem setup. The start and target regions are L/2 in diameter and located at (3L/2, L/2) and (L/2, L/2), respectively. b A naive policy achieves 40.9% success rate on average. c The velocity RL swimmer trained on the cylinder wake navigates the double gyre flow poorly, indicating its navigation policy did not generalize. d After receiving training for the double gyre flow, the velocity RL swimmer is able to adapt and navigate more effectively than either swimmer. As with the cylinder flow, successful attempts to reach the target are green, while unsuccessful attempts are red. An episode is successful when a swimmer reaches within a radius of L/50 around the target location. The stated success rates are averaged over 12,500 episodes and are shown with one standard deviation arising from the five times each swimmer was trained.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 2

  1. Novati, G. & Koumoutsakos, P. Remember and Forget for Experience Replay.arXiv:1807.05827 [cs, stat] (2019).
  2. Sutton, R. S., McAllester, D., Singh, S. & Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, vol. 12 (eds. Solla, S., Leen, T. & Müller, K.) (MIT Press, 2000).

Cited by

See all 108

Metadata

Authors
Gunnarson, P., Mandralis, I., Novati, G., Koumoutsakos, P., & Dabiri, J. O.
Affiliations
  • California Institute of Technology (US)
  • ETH Zurich (CH)
  • Harvard University (US)
Corresponding author
Dabiri, John O.
Venue
Nature Communications
Published
2021-12-08
Volume
12
Issue
1
Pages
7143-7143
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3130787521
Cited by
108
References
33
Citations by year
2026: 21 · 2025: 34 · 2024: 20 · 2023: 17 · 2022: 15 · 2021: 1
FWCI
4.64
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • CaltechAUTHORS (California Institute of Technology) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • CaltechAUTHORS (California Institute of Technology) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • ETH Zürich Research Collection (cc-by)
Funders
National Science Foundation (DGE‐1745301)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Aerospace Engineering › Biomimetic flight and propulsion mechanisms
  • Physical Sciences › Engineering › Ocean Engineering › Maritime Navigation and Safety
Keywords
vortex flow, point-to-point navigation, reinforcement learning, underwater robots, velocity sensing, experience replay, unsteady flow field

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning efficient navigation in vortical flow fields. https://app.cochat.ai/papers/10.1038/s41467-021-27015-y
The paper
Gunnarson, P., Mandralis, I., Novati, G., Koumoutsakos, P., & Dabiri, J. O. (2021). Learning efficient navigation in vortical flow fields. Nature Communications. https://doi.org/10.1038/s41467-021-27015-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.