Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks
Open accessAbstract
In this work, we present a novel gaze-assisted natural language processing (NLP)-based video captioning model to describe routine second-trimester fetal ultrasound scan videos in a vocabulary of spoken sonography. The primary novelty of our multi-modal approach is that the learned video captioning model is built using a combination of ultrasound video, tracked gaze and textual transcriptions from speech recordings. The textual captions that describe the spatio-temporal scan video content are learnt from sonographer speech recordings. The generation of captions is assisted by sonographer gaze-tracking information reflecting their visual attention while performing live-imaging and interpreting a frozen image. To evaluate the effect of adding, or withholding, different forms of gaze on the video model, we compare spatio-temporal deep networks trained using three multi-modal configurations, namely: (1) a gaze-less neural network with only text and video as input, (2) a neural network additionally using real sonographer gaze in the form of attention maps, and (3) a neural network using automatically-predicted gaze in the form of saliency maps instead. We assess algorithm performance through established general text-based metrics (BLEU, ROUGE-L, F1 score), a domain-specific metric (ARS), and metrics that consider the richness and efficiency of the generated captions with respect to the scan video. Results show that the proposed gaze-assisted models can generate richer and more diverse captions for clinical fetal ultrasound scan videos than those without gaze at the expense of the perceived sentence structure. The results also show that the generated captions are similar to sonographer speech in terms of discussing the visual content and the scanning actions performed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Alsharid et al. developed a novel video captioning model for the description of second-trimester US scans by training an AI model with speech recordings and gaze-tracking information of sonographers while performing US scans.
“Lastly, Alsharid et al. developed a novel video captioning model for the description of second-trimester US scans by training an AI model with speech recordings and gaze-tracking information of sonographers while performing US scans [159].”
AI models have been designed for image acquisition and classification, but also for facilitating or omitting repetitive work-intense tasks such as scan report production or captioning of US videos.
“AI models have been designed for image acquisition and classification, but also for facilitating or omitting repetitive work-intense tasks such as scan report production or captioning of US videos [140,159].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 81 · Cited by 25
Research works printed but not linked: 20
- M. Alsharid et al. In: Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis. Springer, pp. 75-84.
- Alsharid, M., Sharma, H., Drukker, L., Chatelain, P., Papageorghiou, A.T., Noble, J.A., 2019. Captioning ultrasound images automatically. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 338-346.
- Bird, S., Klein, E., Loper, E., 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O'Reilly Media, Inc..
- Cai, Y., 2019. Deep Learning Sonographer Visual Attention (Ph.D. thesis). University of Oxford.
- Cai, Y., et al., 2018a. Multi-task SonoEyeNet: detection of fetal standardized planes assisted by generated sonographer attention maps. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 871-879.
- Cai, Y., et al., 2018b. Sonoeyenet: Standardized fetal ultrasound plane detection in- formed by eye tracking. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, pp. 1475-1478.
- Cuturi, M., Blondel, M., 2017. Soft-DTW: a differentiable loss function for time-series. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, pp. 894-903.
- Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Molti- santi, D., Munro, J., Perrett, T., Price, W., et al., 2018. Scaling egocentric vision: The epic-kitchens dataset. In: Proceedings of the European Conference on Computer Vision. ECCV, pp. 720-736.
- Droste, R., Cai, Y., Sharma, H., Chatelain, P., Papageorghiou, A.T., Noble, J.A., 2019. Towards capturing sonographic experience: cognition-inspired ultrasound video saliency prediction. In: Annual Conference on Medical Image Understanding and Analysis. Springer, pp. 174-186.
- Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D., 2010. Every picture tells a story: Generating sentences from images. In: European Conference on Computer Vision. Springer, pp. 15-29.
- Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y., 2016. Deep Learning. Vol. 1, (2), MIT press Cambridge.
- Ba J. Adam: a method for stochastic optimization. In: The International Conference on Learning Representations.
- Lin, C.-Y., 2004. Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74-81.
- Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L., 2014. Microsoft coco: Common objects in context. In: European Conference on Computer Vision. Springer, pp. 740-755.
- Ordonez, V., Kulkarni, G., Berg, T.L., 2011. Im2text: Describing images using 1 million captioned photographs. In: Advances in Neural Information Processing Systems. pp. 1143-1151.
- Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Pinkal, M., Schiele, B., 2014. Coherent multi-sentence video description with variable level of detail. In: German Conference on Pattern Recognition. Springer, pp. 184-195.
- Saab, K., Hooper, S.M., Sohoni, N.S., Parmar, J., Pogatchnik, B., Wu, S., Dunnmon, J.A., Zhang, H.R., Rubin, D., Ré, C., 2021. Observational supervision for medical image classification using gaze data. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 603-614.
- Sharma, H., Droste, R., Chatelain, P., Drukker, L., Papageorghiou, A., Noble, J., 2019. Spatio-temporal partitioning and description of full-length routine fetal anomaly ultrasound scans. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, pp. 987-990.
- Xiong, Y., Du, B., Yan, P., 2019. Reinforced transformer for medical image captioning. In: International Workshop on Machine Learning in Medical Imaging. Springer, pp. 673-680.
- Xu, R., Xiong, C., Chen, W., Corso, J.J., 2015b. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In: AAAI. Vol. 5, Citeseer, p.
- Authors
- Alsharid, M., Cai, Y., Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A.
- Affiliations
- Khalifa University of Science and Technology (AE)
- University of Oxford (GB)
- Tel Aviv University (IL)
- Corresponding author
- Alsharid, Mohammad
- ORCID
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2022-11
- Volume
- 82
- Pages
- 102630
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W4296177022
- Cited by
- 25
- References
- 81
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 13 · 2023: 2
- FWCI
- 1.64
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Medical Image Analysis (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute for Health and Care Research
- Engineering and Physical Sciences Research Council (EP/M013774/1, EP/MO13774/1, EP/T028572/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- prenatal ultrasound, video captioning, gaze tracking, multimodal deep learning, attention maps, saliency maps, second-trimester ultrasound, clinical text generation
- MeSH terms
- Algorithms, Female, Humans, Pregnancy, Ultrasonography, Prenatal, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
