CoChat
Sign Up
Cited by 241Underwater Acoustics ResearchAdvanced Neural Network ApplicationsUnderwater Vehicles and Communication Systems

Underwater object classification in sidescan sonar images using deep transfer learning and semisynthetic training data

Open access
Huo, Guanying, Wu, Ziyin, Li, Jiabiao·2020·IEEE Access
DOI 10.1109/access.2020.2978880
Open this paper in CoChat — free accountOpen PDF

Abstract

Sidescan sonars are increasingly used in underwater search and rescue for drowning victims, wrecks and airplanes. Automatic object classification or detection methods can help a lot in case of long searches, where sonar operators may feel exhausted and therefore miss the possible object. However, most of the existing underwater object detection methods for sidescan sonar images are aimed at detecting mine-like objects, ignoring the classification of civilian objects, mainly due to lack of dataset. So, in this study, we focus on the multi-class classification of drowning victim, wreck, airplane, mine and seafloor in sonar images. Firstly, through a long-term accumulation, we built a real sidescan sonar image dataset named SeabedObjects-KLSG, which currently contains 385 wreck, 36 drowning victim, 62 airplane, 129 mine and 578 seafloor images. Secondly, considering the real dataset is imbalanced, we proposed a semisynthetic data generation method for producing sonar images of airplanes and drowning victims, which uses optical images as input, and combines image segmentation with intensity distribution simulation of different regions. Finally, we demonstrate that by transferring a pre-trained deep convolutional neural network (CNN), e.g. VGG19, and fine-tuning the deep CNN using 70% of the real dataset and the semisynthetic data for training, the overall accuracy on the remaining 30% of the real dataset can be eventually improved to 97.76%, which is the highest among all the methods. Our work indicates that the combination of semisynthetic data generation and deep transfer learning is an effective way to improve the accuracy of underwater object classification.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2020.2978880

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Much work in underwater acoustic data classification is conducted using synthetic data only or on a limited set of real data augmented with semisynthetic examples for training.

    “Therefore, much work in this area is conducted using synthetic data only [124], or on a limited set of real data augmented with semisynthetic examples for training [62,128,139].”

    From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [128]

  • Transfer learning has been used to create an automatic, multiclass object classifier using side-scan sonar data for applications including detection of sunken ship wrecks and aircraft, drowning victims, and mine-like objects.

    “TL has been used to create an automatic, multiclass object classifier using data from side-scan sonars for a range of applications, including the detection of sunken ship wrecks and aircraft, drowning victims, and MLOs [128].”

    From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [128]

  • A transfer-learning CNN based on VGG-19 pretrained with ImageNet and fine-tuned with a semisynthetic sonar dataset showed better performance and accuracy than Support Vector Machines, a shallow CNN, and Deep Forests.

    “The results reported in [128] were compared with more traditional methods for object classification, such as Support Vector Machines, a shallow CNN [134] and Deep Forests [135], where the TL-CNN showed better performance and accuracy.”

    From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [128]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. T. Celik and T. Tjahjadi, ''A novel method for sidescan sonar image seg- mentation,'' IEEE J. Ocean. Eng., vol. 36, no. 2, pp. 186-194, Apr. 2011.
  2. S. Reed, Y. Petillot, and J. Bell, ''An automatic approach to the detection and extraction of mine features in sidescan sonar,'' IEEE J. Ocean. Eng., vol. 28, no. 1, pp. 90-105, Jan. 2003.
  3. C. Barngrover, R. Kastner, and S. Belongie, ''Semisynthetic versus real-world sonar training data for the classification of mine-like objects,'' IEEE J. Ocean. Eng., vol. 40, no. 1, pp. 48-56, Jan. 2015.
  4. X.-F. Ye, Z.-H. Zhang, P. X. Liu, and H.-L. Guan, ''Sonar image segmen- tation based on GMRF and level-set models,'' Ocean Eng., vol. 37, no. 10, pp. 891-901, Jul. 2010.
  5. M. Mignotte, C. Collet, P. Pérez, and P. Bouthemy, ''Three-class Markovian segmentation of high-resolution sonar images,'' Comput. Vis. Image Understand., vol. 76, no. 3, pp. 191-204, Dec. 1999.
  6. M. Mignotte, C. Collet, P. Perez, and P. Bouthemy, ''Sonar image segmen- tation using an unsupervised hierarchical MRF model,'' IEEE Trans. Image Process., vol. 9, no. 7, pp. 1216-1231, Jul. 2000.
  7. G. G. Acosta and S. A. Villar, ''Accumulated CA-CFAR process in 2-D for online object detection from sidescan sonar data,'' IEEE J. Ocean. Eng., vol. 40, no. 3, pp. 558-569, Jul. 2015.
  8. M. Lianantonakis and Y. R. Petillot, ''Sidescan sonar segmentation using texture descriptors and active contours,'' IEEE J. Ocean. Eng., vol. 32, no. 3, pp. 744-752, Jul. 2007.
  9. S. Daniel, S. Guillaudeux, and E. Maillard, ''Adaptation of a partial shape recognition approach,'' in Proc. IEEE Int. Conf. Syst., Man, Cybern. Com- put. Cybern. Simulation, Oct. 1997, pp. 2157-2162.
  10. G. Huo, S. X. Yang, Q. Li, and Y. Zhou, ''A robust and fast method for sidescan sonar image segmentation using nonlocal despeckling and active contour model,'' IEEE Trans. Cybern., vol. 47, no. 4, pp. 855-872, Apr. 2017.
  11. A. Abu and R. Diamant, ''A statistically-based method for the detection of underwater objects in sonar imagery,'' IEEE Sensors J., vol. 19, no. 16, pp. 6858-6871, Aug. 2019.
  12. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, U.K.: MIT Press, 2016.
  13. S. Daniel, F. Le Leannec, C. Roux, B. Soliman, and E. P. Maillard, ''Side-scan sonar image matching,'' IEEE J. Ocean. Eng., vol. 23, no. 3, pp. 245-259, Jul. 1998.
  14. G. J. Dobeck, J. C. Hyland, and L. Smedley, ''Automated detection and classification of sea mines in sonar imagery,'' Proc. SPIE, vol. 3079, pp. 90-110, Jul. 1997.
  15. H. Midelfart, J. Groen, and Ø. Midtgaard, ''Template matching methods for object classification in synthetic aperture sonar images,'' in Proc. Underwater Acoust. Meas. Conf., 2009, pp. 1-6.
  16. V. Myers and J. Fawcett, ''A template matching procedure for automatic target recognition in synthetic aperture sonar imagery,'' IEEE Signal Pro- cess. Lett., vol. 17, no. 7, pp. 683-686, Jul. 2010.
  17. H. Cho, J. Gu, and S.-C. Yu, ''Robust sonar-based underwater object recognition against angle-of-view variation,'' IEEE Sensors J., vol. 16, no. 4, pp. 1013-1025, Feb. 2016.
  18. J. Sawas, Y. Petillot, and Y. Pailhas, ''Cascade of boosted classifiers for rapid detection of underwater objects,'' in Proc. Eur. Conf. Underwater Acoust., vol. 10, 2010, pp. 1-8.
  19. C. Barngrover, A. Althoff, P. DeGuzman, and R. Kastner, ''A brain-computer interface (BCI) for the detection of mine-like objects in sidescan sonar imagery,'' IEEE J. Ocean. Eng., vol. 41, no. 1, pp. 123-138, Jan. 2016.
  20. P. Hollesen, W. A. Connors, and T. Trappenberg, ''Comparison of learned versus engineered features for classification of mine like objects from raw sonar images,'' in Advances in Artificial Intelligence (Lecture Notes in Computer Science), vol. 6657. Berlin, Germany: Springer-Verlag, 2011, pp. 174-185.
  21. D. P. Williams, V. Myers, and M. Schatten Silvious, ''Mine classification with imbalanced data,'' IEEE Geosci. Remote Sens. Lett., vol. 6, no. 3, pp. 528-532, Jul. 2009.
  22. X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, ''Deep learning in remote sensing: A comprehensive review and list of resources,'' IEEE Geosci. Remote Sens. Mag., vol. 5, no. 4, pp. 8-36, Dec. 2017.
  23. G. E. Hinton, S. Osindero, and Y.-W. Teh, ''A fast learning algorithm for deep belief nets,'' Neural Comput., vol. 18, no. 7, pp. 1527-1554, Jul. 2006.
  24. K. Simonyan and A. Zisserman, ''Very deep convolutional networks for large-scale image recognition,'' 2014, arXiv:1409.1556. [Online]. Avail- able: http://arxiv.org/abs/1409.1556 · arXiv
  25. R. Girshick, J. Donahue, T. Darrell, and J. Malik, ''Region-based convolu- tional networks for accurate object detection and semantic segmentation,'' IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 1, pp. 142-158, Jan. 2016.
  26. J. Long, E. Shelhamer, and T. Darrell, ''Fully convolutional networks for semantic segmentation,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3431-3440.
  27. J. Fu, J. Liu, Y. Wang, and H. Lu, ''Densely connected deconvolutional network for semantic segmentation,'' in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2017, pp. 1520-1528.
  28. D. Einsidler, M. Dhanak, and P.-P. Beaujean, ''A deep learning approach to target recognition in side-scan sonar imagery,'' in Proc. OCEANS MTS/IEEE Charleston, Charleston, SC, USA, Oct. 2018, pp. 1-4.
  29. X. Cao, R. Togneri, X. Zhang, and Y. Yu, ''Convolutional neural network with second-order pooling for underwater target classification,'' IEEE Sensors J., vol. 19, no. 8, pp. 3058-3066, Apr. 2019.
  30. P. Zhu, J. Isaacs, B. Fu, and S. Ferrari, ''Deep learning feature extraction for target recognition and classification in underwater sonar images,'' in Proc. IEEE 56th Annu. Conf. Decis. Control (CDC), Melbourne, VIC, Australia, Dec. 2017, pp. 2724-2731.
  31. K. Denos, M. Ravaut, A. Fagette, and H.-S. Lim, ''Deep learning applied to underwater mine warfare,'' in Proc. OCEANS, Aberdeen, Scotland, Jun. 2017, pp. 1-7.
  32. K. Zhu, J. Tian, and H. Huang, ''Underwater object images classification based on convolutional neural network,'' in Proc. IEEE 3rd Int. Conf. Signal Image Process. (ICSIP), Shenzhen, China, Jul. 2018, pp. 301-305.
  33. D. P. Williams, ''Underwater target classification in synthetic aperture sonar imagery using deep convolutional neural networks,'' in Proc. 23rd Int. Conf. Pattern Recognit. (ICPR), Cancún, Mexico, Dec. 2016, pp. 2497-2502.
  34. T. Rimavicius and A. Gelzinis, ''A comparison of the deep learning meth- ods for solving seafloor image classification task,'' Inform. Softw. Technol., vol. 756, pp. 442-453, 2017.
  35. A. Diegues, J. Pinto, P. Ribeiro, R. Frias, and D. C. Alegre, ''Automatic habitat mapping using convolutional neural networks,'' in Proc. IEEE/OES Auto. Underwater Vehicle Workshop (AUV), Nov. 2018, pp. 1-6.
  36. X. Luo, X. Qin, Z. Wu, F. Yang, M. Wang, and J. Shang, ''Sediment classification of small-size seabed acoustic images using convolutional neural networks,'' IEEE Access, vol. 7, pp. 98331-98339, 2019.
  37. Y. Wang, C. Wang, H. Zhang, Y. Dong, and S. Wei, ''A SAR dataset of ship detection for deep learning under complex backgrounds,'' Remote Sens., vol. 11, no. 7, p. 765, 2019.
  38. Y. Wang, C. Wang, H. Zhang, Y. Dong, and S. Wei, ''Automatic ship detection based on RetinaNet using multi-resolution Gaofen-3 imagery,'' Remote Sens., vol. 11, no. 5, p. 531, 2019.
  39. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, ''Gradient-based learn- ing applied to document recognition,'' Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
  40. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, ''Going deeper with convolutions,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1-9.
  41. H. Chang, J. Han, C. Zhong, A. M. Snijders, and J.-H. Mao, ''Unsupervised transfer learning via multi-scale convolutional sparse coding for biomedi- cal applications,'' IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 5, pp. 1182-1194, May 2018.
  42. M. Long, H. Zhu, J. Wang, and M. I. Jordan, ''Unsupervised domain adaptation with residual transfer networks,'' in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2016, pp. 136-144.
  43. O. George and R. Bahl, ''Simulation of backscattering of high frequency sound from complex objects and sand sea-bottom,'' IEEE J. Ocean. Eng., vol. 20, no. 2, pp. 119-130, Apr. 1995.
  44. J. M. Bell, ''Application of optical ray tracing techniques to the simulation of sonar images,'' Opt. Eng., vol. 36, no. 6, pp. 1806-1813, Jun. 1997.
  45. T. Capéran, G. Hayward, and R. Chapman, ''A 3D simulator for the design and evaluation of sonar system instrumentation,'' Meas. Sci. Technol., vol. 10, no. 12, pp. 1116-1126, Dec. 1999.
  46. J. A. Fawcett, ''Modeling of high-frequency scattering from objects using a hybrid Kirchhoff/diffraction approach,'' J. Acoust. Soc. Amer., vol. 109, no. 4, pp. 1312-1319, Apr. 2001.
  47. J. A. Fawcett and R. Lim, ''Evaluation of the integrals of target/seabed scattering using the method of complex images,'' J. Acoust. Soc. Amer., vol. 114, no. 3, pp. 1406-1415, Sep. 2003.
  48. G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray, ''Visual categorization with bags of keypoints,'' in Proc. Workshop Stat. Learn. Comput. Vis. (ECCV), 2004, pp. 1-22.
  49. Z.-H. Zhou and J. Feng, ''Deep forest: Towards an alternative to deep neural networks,'' in Proc. 26th Int. Joint Conf. Artif. Intell., Macao, China, Aug. 2017, pp. 3353-3359.
  50. Y. Wang, C. Wang, and H. Zhang, ''Ship classification in high-resolution SAR images using deep learning of small datasets,'' Sensors, vol. 18, no. 9, p. 2929, 2018.
  51. C.-C. Chang and C.-J. Lin, ''LIBSVM: A library for support vector machines,'' ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, pp. 1-27, Apr. 2011.
  52. E. Tzeng, J. Hoffman, N. Zhang, and K. Saenko, ''Simultaneous deep transfer across domains and tasks,'' in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2015, pp. 4068-4076.
  53. D. C. Dowson and B. V. Landau, ''The Fréchet distance between multivari- ate normal distributions,'' J. Multivariate Anal., vol. 12, no. 3, pp. 450-455, Sep. 1982.
  54. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, ''SSD: Single shot MultiBox detector,'' in Proc. Eur. Conf. Comput. Vis. (ECCV), 2016, pp. 21-37.
  55. GUANYING HUO (Member, IEEE) received the B.Sc. degree in telecommunications engineering and the M.Sc. degree in information and communi- cations engineering from Xidian University, Xi'an, China, in 2001 and 2004, respectively, and the Ph.D. degree in computer application technology from Hohai University, Nanjing, China, in 2012. He is currently a Postdoctoral with the Second Institute of Oceanography, Ministry of Natural Resources, and an Associate Professor with Hohai University. His current research interests include sonar image processing and deep learning. ZIYIN WU received the Ph.D. degree in marine geology from Zhejiang University, Zhejiang, China, in 2008. He is currently a Professor with the Second Institute of Oceanography, Ministry of Natural Resources and the School of Oceanogra- phy, Shanghai Jiao Tong University. His primary research interests include seabed survey and infor- mation systems.

Cited by

Sorted by citations

Authors
Huo, G., Wu, Z., & Li, J.
Affiliations
  • Hohai University (CN)
  • Ministry of Natural Resources (CN)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2020
Volume
8
Pages
47407-47418
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2020.2978880
OpenAlex
W3009376633
Cited by
241
References
11
Citations by year
2026: 41 · 2025: 58 · 2024: 49 · 2023: 36 · 2022: 33 · 2021: 21 · 2020: 3
FWCI
11.16
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Natural Science Foundation of China (41830540, 41876097, 41576099)
  • Scientific Research Fund of the Second Institute of Oceanography, Ministry of Natural Resources (JB1803)
Topics
  • Physical Sciences › Earth and Planetary Sciences › Oceanography › Underwater Acoustics Research
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Ocean Engineering › Underwater Vehicles and Communication Systems
Keywords
underwater target classification, underwater search and rescue, landmine detection, transfer learning, class imbalance, image segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review