Self-Supervised Contrastive Video-Speech Representation Learning for Ultrasound
Open accessAbstract
In medical imaging, manual annotations can be expensive to acquire and sometimes infeasible to access, making conventional deep learning-based models difficult to scale. As a result, it would be beneficial if useful representations could be derived from raw data without the need for manual annotations. In this paper, we propose to address the problem of self-supervised representation learning with multi-modal ultrasound video-speech raw data. For this case, we assume that there is a high correlation between the ultrasound video and the corresponding narrative speech audio of the sonographer. In order to learn meaningful representations, the model needs to identify such correlation and at the same time understand the underlying anatomical features. We designed a framework to model the correspondence between video and audio without any kind of human annotations. Within this framework, we introduce cross-modal contrastive learning and an affinity-aware self-paced learning scheme to enhance correlation modelling. Experimental evaluations on multi-modal fetal ultrasound video and audio show that the proposed approach is able to learn strong representations and transfers well to downstream tasks of standard plane detection and eye-gaze prediction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The innate relationship tasks in combined studies range from augmentation prediction and patch positioning prediction, rotation prediction, and ultrasound video to speech correspondence prediction.
“The innate relationship tasks range from augmentation prediction and patch positioning prediction69, rotation prediction70, and ultrasound video to speech correspondence prediction53.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [53]
Jiao et al. leveraged the correlation between fetal ultrasonography and the narrative speech of the sonographer to create a pre-text task for self-supervision, and subsequently used the learned representations for downstream standard plane classification on sonograms.
“Jiao et al. cleverly leveraged the correlation between fetal ultrasonography and the narrative speech of the sonographer to create a pre-text task for self-supervision, and subsequently used the learned representations for downstream standard plane classification on sonograms53.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157 · see citing works
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Arandjelović, R., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 836 · see citing works
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A. B., & Durand, F. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2815601 · Cited by 835 · see citing works
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704 · see citing works
Unsupervised Representation Learning by Sorting Sequences
Lee, H.-Y., Huang, J., Singh, M., & Yang, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.79 · Cited by 529 · see citing works
SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound
Baumgartner, C. F., Kamnitsas, K., Matthew, J., Fletcher, T. P., Smith, S., Koch, L. M., Kainz, B., & Rueckert, D. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2712367 · Cited by 416 · see citing works
Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik’s Cube
Zhuang, X., Li, Y., Hu, Y., Ma, K., Yang, Y., & Zheng, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_46 · Cited by 153 · see citing works
Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese Networks
Spitzer, H., Kiwitz, K., Amunts, K., Stefan Harmeling, B., & Dickscheid, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00931-1_76 · Cited by 77 · see citing works
Self-Supervised Representation Learning for Ultrasound Video
Jiao, J., Droste, R., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098666 · Cited by 56 · see citing works
Self-citationApproach for Energy-Based Voice Detector with Adaptive Scaling Factor
Sakhnov, K., Verteletskaya, E., & Šimák, B. (2009) · Cited by 30 · see citing works
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [4]Chen T Kornblith S Norouzi M Hinton G A simple framework for contrastive learning of visual representations ICML 2020
- [6]He K Fan H Wu Y Xie S Girshick R Momentum contrast for unsupervised visual representation learning CVPR 2020
- [7]Hénaff OJ Srinivas A De Fauw J Razavi A Doersch C Eslami S Oord Avd Data-efficient image recognition with contrastive predictive coding 2019 arXiv preprint arXiv:1905.09272 · arXiv
- [11]Oord Avd Li Y Vinyals O Representation learning with contrastive predictive coding 2018 arXiv preprint arXiv:1807.03748 · arXiv
Cited by
Sorted by citations
Contrastive Representation Learning: A Framework and Review
Le-Khac, P. H., Healy, G., & Smeaton, A. F. (2020) · IEEE Access · doi:10.1109/access.2020.3031549 · Cited by 862 · see citing works
Self-supervised learning in medicine and healthcare
Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605 · see citing works
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479 · see citing works
“…et al. pre-trained their models simultaneously with two innate relationship pre-text tasks (slice order prediction and geometric transformation prediction), and showed that a weight-sharing Siamese network out-performs a single disentanged model for combining the two pre-training objectives 53 .”
BackgroundMethodAudio self-supervised learning: A survey
Liu, S., Mallol-Ragolta, A., Parada‐Cabaleiro, E., Qian, K., Jing, X., Kathan, A., Hu, B., & Schuller, B. W. (2022) · Patterns · doi:10.1016/j.patter.2022.100616 · Cited by 116 · see citing works
Positional Contrastive Learning for Volumetric Medical Image Segmentation
Zeng, D., Wu, Y., Hu, X., Xu, X., Yuan, H., Huang, M., Zhuang, J., Hu, J., & Shi, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_21 · Cited by 93 · see citing works
Dissecting self-supervised learning methods for surgical computer vision
Ramesh, S., Srivastav, V., Alapatt, D., Yu, T., Murali, A., Sestini, L., Nwoye, C. I., Hamoud, I., Sharma, S., Fleurentin, A., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102844 · Cited by 70 · see citing works
Contrastive self-supervised learning from 100 million medical images with optional supervision
Ghesu, F. C., Georgescu, B., Mansoor, A., Yoo, Y., Neumann, D., Patel, P. B., Vishwanath, R. S., Balter, J., Cao, Y., Grbic, S., et al. (2022) · Journal of Medical Imaging · doi:10.1117/1.jmi.9.6.064503 · Cited by 50 · see citing works
A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
Wang, W.-C., Ahn, E., Feng, D.-W., & Kim, J. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1406-4 · Cited by 59 · see citing works
Chen, Y., Zhang, C., Liu, L., Feng, C., Dong, C., Luo, Y., & Wan, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87237-3_60 · Cited by 56 · see citing works
Lesion-Based Contrastive Learning for Diabetic Retinopathy Grading from Fundus Images
Huang, Y., Lin, L., Cheng, P., Lyu, J., & Tang, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_11 · Cited by 51 · see citing works
An Improved Inter-Intra Contrastive Learning Framework on Self-Supervised Video Representation
Tao, L., Wang, X., & Yamasaki, T. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3141051 · Cited by 50 · see citing works
Extracting keyframes of breast ultrasound video using deep reinforcement learning
Huang, R., Ying, Q., Lin, Z., Zheng, Z., Tan, L., Tang, G., Zhang, Q., Luo, M., Yi, X., Liu, P., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102490 · Cited by 50 · see citing works
P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene UnderstandingPreprint
Liu, Y., Yi, L., Zhang, S., Fan, Q., Funkhouser, T. A., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13089 · Cited by 33 · see citing works
Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound
Fu, Z., Jiao, J., Yasrab, R., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25066-8_23 · Cited by 24 · see citing works
Self-citationWang, B., Li, Q., & You, Z. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.12.028 · Cited by 22 · see citing works
Self-Supervised Interactive Embedding for One-Shot Organ Segmentation
Yang, Y., Wang, B., Zhang, D., Yuan, Y., Yan, Q., Zhao, S., You, Z., & Han, J. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3265033 · Cited by 22 · see citing works
Voice-Assisted Image Labeling for Endoscopic Ultrasound Classification Using Neural Networks
Bonmati, E., Hu, Y., Grimwood, A., Johnson, G., Goodchild, G., Keane, M. G., Gurusamy, K. S., Davidson, B. R., Clarkson, M. J., Pereira, S. P., et al. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3139023 · Cited by 21 · see citing works
Generating and Weighting Semantically Consistent Sample Pairs for Ultrasound Contrastive Learning
Chen, Y., Zhang, C., Ding, C. H. Q., & Liu, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3228254 · Cited by 20 · see citing works
Zhao, Z., & Yang, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_24 · Cited by 17 · see citing works
SWaCo: Safe Wafer Bin Map Classification With Self-Supervised Contrastive Learning
Kwak, M. G., Lee, Y. J., & Kim, S. B. (2023) · IEEE Transactions on Semiconductor Manufacturing · doi:10.1109/tsm.2023.3280891 · Cited by 15 · see citing works
HiCo: Hierarchical Contrastive Learning for Ultrasound Video Model Pretraining
Zhang, C., Chen, Y., Liu, L., Liu, Q., & Zhou, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26351-4_1 · Cited by 13 · see citing works
Contrastive Learning for View Classification of Echocardiograms
Chartsias, A., Gao, S., Mumith, A., Oliveira, J., Bhatia, K. K., Kainz, B., & Beqiri, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87583-1_15 · Cited by 13 · see citing works
Multimodal Foundation Models for Medical Imaging - A Systematic Review and Implementation GuidelinesPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2024) · medRxiv · doi:10.1101/2024.10.23.24316003 · Cited by 15 · see citing works
Task model-specific operator skill assessment in routine fetal ultrasound scanning
Wang, Y., Yang, Q., Drukker, L., Papageorghiou, A., Hu, Y., & Noble, J. A. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02642-y · Cited by 12 · see citing works
Self-citationAudio–visual self-supervised representation learning: A survey
Alsuwat, M. S., Al-Shareef, S., & Alghamdi, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129750 · Cited by 7 · see citing works
Self-Supervised Contrastive Learning to Predict Alzheimer’s Disease Progression with 3D Amyloid-PETPreprint
Kwak, M. G., Su, Y., Chen, K., Weidman, D. A., Wu, T., Lure, F. Y. M., & Li, J. (2023) · medRxiv · doi:10.1101/2023.04.20.23288886 · Cited by 6 · see citing works
Fang, S., Hong, S., Li, Q., Li, P., Coats, T., Zou, B., & Kong, G. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105680 · Cited by 5 · see citing works
Liu, C., Zeng, Y., Liu, Q., Song, K., Kong, B., & Su, X. (2024) · Biomedical Optics Express · doi:10.1364/boe.510022 · Cited by 5 · see citing works
Visually-aware Acoustic Event Detection using Heterogeneous Graphs
Shirian, A., Somandepalli, K., Sánchez, V., & Guha, T. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10670 · Cited by 4 · see citing works
Deep clustering for abdominal organ classification in ultrasound imaging
Dadoun, H., Delingette, H., Rousseau, A.-L., de Kerviler, É., & Ayache, N. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.3.034502 · Cited by 3 · see citing works
Anukem, S., Ozougwu, L., Ataguba, G., Milos, E., & Orji, R. (2025) · SeGAH ... IEEE ... International Conference on Serious Games and Applications for Health/IEEE International Conference on Serious Games and Applications for Health · doi:10.1109/segah65397.2025.11168427 · Cited by 2 · see citing works
Challier, C., Sun, X., Mahendiran, T., Senouf, O. Y., Bruyne, B. D., Auberson, D., Müller, O., Fournier, S., Frossard, P., Abbe, E. A., et al. (2025) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc58623.2025.11253755 · Cited by 3 · see citing works
Audio-visual modelling in a clinical setting
Jiao, J., Alsharid, M., Drukker, L., Papageorghiou, A. T., Zisserman, A., & Noble, J. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-66160-4 · Cited by 2 · see citing works
Self-citationDual Representation Learning From Fetal Ultrasound Video and Sonographer Audio
Gridach, M., Alsharid, M., Jiao, J., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635693 · Cited by 2 · see citing works
Self-citationEffective Sample Pair Generation for Ultrasound Video Contrastive Representation Learning.Preprint
Chen, Y., Zhang, C., Liu, L., Feng, C., Dong, C., Luo, Y., & Wan, X. (2020) · arXiv (Cornell University) · Cited by 2 · see citing works
Chen, Y., Zhang, C., Liu, L., Feng, C., Dong, C., Luo, Y., & Wan, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13066 · Cited by 2 · see citing works
Voice-assisted multimodal fusion network for difficult airway assessment
Li, X., Zhang, G., Ye, W., Peng, B., & Yao, Y. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-03916-z · Cited by 1 · see citing works
Kumari, P., & Meena, R. (2026) · Cureus · doi:10.7759/cureus.114172
- Authors
- Jiao, J., Cai, Y., Alsharid, M., Drukker, L., Papageorghiou, A. T., & Noble, J. A.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Jiao, Jianbo
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2020
- Volume
- 12263
- Pages
- 534-543
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- PMC
- PMC7116253
- OpenAlex
- W3094123278
- Cited by
- 38
- References
- 18
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 5 · 2023: 9 · 2022: 9 · 2021: 6 · 2020: 4
- FWCI
- 2.14
- Type
- conference paper
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Lecture notes in computer science (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Engineering and Physical Sciences Research Council (EP/M013774/1, EP/M013774/)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Ultrasound in Clinical Applications
- Keywords
- self-supervised representation learning, contrastive learning, ultrasound video, cross-modal learning, prenatal ultrasound, annotation-free learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
