Multi-Modal Learning from Video, Eye Tracking, and Pupillometry for Operator Skill Characterization in Clinical Fetal Ultrasound
Open accessAbstract
This paper presents a novel multi-modal learning approach for automated skill characterization of obstetric ultrasound operators using heterogeneous spatio-temporal sensory cues, namely, scan video, eye-tracking data, and pupillometric data, acquired in the clinical environment. We address pertinent challenges such as combining heterogeneous, small-scale and variable-length sequential datasets, to learn deep convolutional neural networks in real-world scenarios. We propose spatial encoding for multi-modal analysis using sonography standard plane images, spatial gaze maps, gaze trajectory images, and pupillary response images. We present and compare five multi-modal learning network architectures using late, intermediate, hybrid, and tensor fusion. We build models for the Heart and the Brain scanning tasks, and performance evaluation suggests that multi-modal learning networks outperform uni-modal networks, with the best-performing model achieving accuracies of 82.4% (Brain task) and 76.4% (Heart task) for the operator skill classification problem.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI in scan workflow enabled transfer learning for US novices.
“enabling transfer learning for US novices [160]”
From Evolving the Era of 5D Ultrasound? A Systematic Literature Review on the Applications for Artificial Intelligence Ultrasound Imaging in Obstetrics and Gynecology (2023) · cites this paper as [160]
The Introduction and Methods (page numbers unavailable) say the authors used transfer learning to train CNNs for ultrasound operator skill characterization. They do not say that transfer learning was enabled for ultrasound novices or that novices used it to learn scanning. (LLM, conf=0.86)
Experimental — accuracy not yet measuredAI can support US novices in skill training and enables non-experts the acquisition of US images.
“It can support US novices in skill training and enables non-experts the acquisition of US images [160]”
From Evolving the Era of 5D Ultrasound? A Systematic Literature Review on the Applications for Artificial Intelligence Ultrasound Imaging in Obstetrics and Gynecology (2023) · cites this paper as [160]
The Introduction says ultrasound skill assessment can form part of initial training and describes AI models that classify operator skill, which supports a possible training application. It does not show that the models train novices or enable non-experts to acquire ultrasound images; the Methods study scans by newly qualified and experienced operators instead. (LLM, conf=0.80)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works
Pupil dilation as an index of effort in cognitive control tasks: A review
van der Wel, P., & van Steenbergen, H. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1432-y · Cited by 939 · see citing works
Imaging Time-Series to Improve Classification and ImputationPreprint
Wang, Z., & Oates, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00327 · Cited by 715 · see citing works
Surgical data science for next-generation interventions
Maier‐Hein, L., Vedula, S. S., Speidel, S., Navab, N., Kikinis, R., Park, A., Eisenmann, M., Feußner, H., Forestier, G., Giannarou, S., et al. (2017) · Nature Biomedical Engineering · doi:10.1038/s41551-017-0132-7 · Cited by 616 · see citing works
What Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588 · see citing works
Preprocessing pupil size data: Guidelines and code
Kret, M. E., & Sjak-Shie, E. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1075-y · Cited by 476 · see citing works
Drew, T. W., Evans, K. K., Võ, M. L. ‐., Jacobson, F. L., & Wolfe, J. M. (2012) · Radiographics · doi:10.1148/rg.331125023 · Cited by 182 · see citing works
Sharma, H., Drukker, L., Chatelain, P., Droste, R., Papageorghiou, A. T., & Noble, J. A. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101973 · Cited by 48 · see citing works
Self-citationSpatio-temporal visual attention modelling of standard biometry plane-finding navigation
Cai, Y., Droste, R., Sharma, H., Chatelain, P., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101762 · Cited by 41 · see citing works
Self-citationEvaluation of Gaze Tracking Calibration for Longitudinal Biomedical Imaging Studies
Chatelain, P., Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2866274 · Cited by 29 · see citing works
Self-citationSpatio-Temporal Partitioning And Description Of Full-Length Routine Fetal Anomaly Ultrasound Scans
Sharma, H., Droste, R., Chatelain, P., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759149 · Cited by 29 · see citing works
Self-citationPUPILLOMETRY AS A TOOL TO STUDY EXPERTISE IN MEDICINE
Szulewski, A., Kelton, D., & Howes, D. W. (2017) · Frontline Learning Research · doi:10.14786/flr.v5i3.256 · Cited by 22 · see citing works
Differentiating Operator Skill During Routine Fetal Ultrasound Scanning Using Probe Motion Tracking
Wang, Y., Droste, R., Jiao, J., Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60334-2_18 · Cited by 13 · see citing works
Self-citationSharma, H., Drukker, L., Droste, R., et al. (2020) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.22266 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Jost, E., Kosian, P., Jimenez Cruz, J., Albarqouni, S., Gembruch, U., Strizek, B., & Recker, F. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12216833 · Cited by 44 · see citing works
“It can support US novices in skill training and enables non-experts the acquisition of US images [160], e.g., for telehealth approaches in times of shortage of expert sonographers [50].”
BackgroundTask model-specific operator skill assessment in routine fetal ultrasound scanning
Wang, Y., Yang, Q., Drukker, L., Papageorghiou, A., Hu, Y., & Noble, J. A. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02642-y · Cited by 12 · see citing works
Self-citation
- Authors
- Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A.
- Affiliations
- University of Oxford (GB)
- Venue
- 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)
- Publisher
- IEEE
- Published
- 2021-04-13
- Volume
- 2021
- Pages
- 1646-1649
- PMC
- PMC7611522
- OpenAlex
- W3163992391
- Cited by
- 15
- References
- 16
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 3 · 2022: 4 · 2021: 2
- FWCI
- 12.80
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- IEEE International Symposium on Biomedical Imaging (ISBI) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/M013774/1)
- Topics
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Ultrasound in Clinical Applications
- Health Sciences › Medicine › Surgery › Surgical Simulation and Training
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- multimodal learning, eye tracking, pupillometry, gaze trajectories, skill classification, tensor fusion, deep convolutional neural networks, prenatal ultrasound
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
