Dynamic robotic tracking of underwater targets using reinforcement learning
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
To realize the potential of autonomous underwater robots that scale up our observational capacity in the ocean, new techniques are needed. Fleets of autonomous robots could be used to study complex marine systems and animals with either new imaging configurations or by tracking tagged animals to study their behavior. These activities can then inform and create new policies for community conservation. The role of animal connectivity via active movement of animals represents a major knowledge gap related to the distribution of deep ocean populations. Tracking underwater targets represents a major challenge for observing biological processes in situ, and methods to robustly respond to a changing environment during monitoring missions are needed. Analytical techniques for optimal sensor placement and path planning to locate underwater targets are not straightforward in such cases. The aim of this study was to investigate the use of reinforcement learning as a tool for range-only underwater target-tracking optimization, whose promising capabilities have been demonstrated in terrestrial scenarios. To evaluate its usefulness, a reinforcement learning method was implemented as a path planning system for an autonomous surface vehicle while tracking an underwater mobile target. A complete description of an open-source model, performance metrics in simulated environments, and evaluated algorithms based on more than 15 hours of at-sea field experiments are presented. These efforts demonstrate that deep reinforcement learning is a powerful approach that enhances the abilities of autonomous robots in the ocean and encourages the deployment of algorithms like these for monitoring marine biological systems in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Geocentric policies are not invariant to rotations and translations of the flow field and not suitable for autonomous deployment in the open ocean without continuous support from and communication with a terrestrial control center8.”
From Sensing flow gradients is necessary for learning autonomous underwater navigation (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our analysis of the sensory requirements for autonomous underwater navigation (Table 1) indicates that egocentric sensing in the agent’s own world eliminates potential time delays and computations inherent to assessing inertial signals8,83 at the expense of requiring more sensors to observe spatial variations in the flow field at the scale of the navigator.”
From Sensing flow gradients is necessary for learning autonomous underwater navigation (2025) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“The use of deep Q-learning and off-policy algorithms in continuous state and action space presents a major challenge for stability and convergence ( 38 ).”
MethodMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 1×
“Whereas most of the attention in deep RL has focused on game theory (for example to solve Atari games ( 24 ) or to master the game of Go ( 25 )), the same principles can be used to solve path planning and trajectory optimization problems (Table 1).”
BackgroundObject Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545
The State of World Fisheries and Aquaculture 2020
(2020) · doi:10.4060/ca9229en · Cited by 3,167
Underwater acoustic communication channels: Propagation models and statistical characterization
Stojanovic, M., & Preisig, J. (2009) · IEEE Communications Magazine · doi:10.1109/mcom.2009.4752682 · Cited by 2,060
Learning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Science Robotics · doi:10.1126/scirobotics.abc5986 · Cited by 1,156
Underwater sensor networks: applications, advances and challenges
Heidemann, J., Stojanovic, M., & Zorzi, M. (2012) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2011.0214 · Cited by 882
Learning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000448343 · Cited by 752
The generation of plankton patchiness by turbulent stirring
Abraham, E. R. (1998) · Nature · doi:10.1038/35361 · Cited by 456 · cited 1×
“Multiple robotic platforms, in coordination with each other, have been used to characterize different oceanographic phenomena, such as fronts and eddies (10, 11).”
MethodAutonomous navigation of stratospheric balloons using reinforcement learning
Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., & Wang, Z. (2020) · Nature · doi:10.1038/s41586-020-2939-8 · Cited by 286
Research works printed but not linked: 5
- R. S. Sutton A. G. Barto Reinforcement Learning: An Introduction (MIT Press ed. 2 2015).
- T. Haarnoja A. Zhou P. Abbeel S. Levine Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor in Proceedings of the 35th International Conference on (ICML 2018) (ICML 2018) vol. 5 pp. 2976–2989.
- J. K. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. Santos, R. Perez, C. Horsch, C. Dieffendahl, N. L. Williams, Y. Lokesh, P. Ravi, PettingZoo: A standard API for multi-agent reinforcement learning. Adv. Neural Inf. Process. Syst. 18, 15032–15043 (2021).
- T. P. Lillicrap J. J. Hunt A. Pritzel N. Heess T. Erez Y. Tassa D. Silver D. Wierstra Continuous control with deep reinforcement learning in 4th International Conference on Learning Representations (ICLR 2016)–Conference Track Proceedings (ICLR 2016).
- K. Katija P. L. D. Roberts J. Daniels A. Lapides K. Barnard M. Risi B. Y. Ranaan B. G. Woodward J. Takahashi in 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE 2021) pp. 859–868.
Cited by
Chu, S., Lin, M., Li, D., Lin, R., & Xiao, S. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.120139 · Cited by 79
Zheng, X., Jiang, Y., Mete, M., Li, J., Watanabe, I., Yamada, T., & Paik, J. K. (2025) · Science Robotics · doi:10.1126/scirobotics.adx1519 · Cited by 48
Dai, H., Zhang, C., Hu, H., Hu, Z., Sun, H., Liu, K., Li, T., Fu, J., Zhao, P., & Yang, H. (2024) · Advanced Science · doi:10.1002/advs.202405276 · Cited by 41
Chun, S., Kawamura, C., Ohkuma, K., & Maki, T. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3364470 · Cited by 31
“Digital Object Identifier 10.1109/LRA.2024.3364470 To compensate for the limitations of biologging, Autonomous Underwater Vehicles (AUVs) have emerged as one of the remote sensing methods [4], [5], [6].”
BackgroundInfluentialA comprehensive review on the evolution of bio-inspired sensors from aquatic creatures
Zhao, Z., Yang, Q., Li, R., Yang, J., Liu, Q., Zhu, B., Weng, C., Liu, W., Hu, P., Ma, L. M., et al. (2024) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2024.102064 · Cited by 33
Metadata
- Authors
- Masmitja, I., Martin, M., O’Reilly, T., Kieft, B., Palomeras, N., Navarro, J., & Katija, K.
- Affiliations
- Institut de Ciències del Mar (ICM), CSIC, Barcelona 95062, Spain.
- Knowledge Engineering and Machine Learning Group, Universitat Politècnica de Catalunya, Barcelona Tech., Barcelona 08034, Spain.
- Monterey Bay Aquarium Research Institute (US)
- Universitat de Girona (ES)
- Institut de Ciències del Mar (ES)
- Venue
- Science Robotics
- Published
- 2023-07-12
- Volume
- 8
- Issue
- 80
- Pages
- eade7811-eade7811
- ISSN
- 2470-9476 (ISSN-L)
- OpenAlex
- W4385286939
- Cited by
- 96
- References
- 59
- Citations by year
- 2026: 29 · 2025: 44 · 2024: 23
- FWCI
- 11.66
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Science Robotics (publishedVersion)
- PubMed (publishedVersion)
- UPCommons institutional repository (Universitat Politècnica de Catalunya) (submittedVersion, cc-by-nc-nd)
- DIGITAL.CSIC (Spanish National Research Council (CSIC)) (submittedVersion)
- Funders
- European Commission (CEX2019-000928-S, 893089)
- Barcelona Supercomputing Center
- Topics
- Physical Sciences › Engineering › Ocean Engineering › Underwater Vehicles and Communication Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Ocean Engineering › Maritime Navigation and Safety
- Keywords
- underwater target tracking, reinforcement learning, autonomous underwater vehicles, unmanned surface vehicles, path planning, deep reinforcement learning, oceanographic monitoring, marine conservation, field experiments
- MeSH terms
- Algorithms, Animals, Learning, Reinforcement, Psychology, Robotics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.