Task model-specific operator skill assessment in routine fetal ultrasound scanning
Open accessAbstract
PURPOSE: For highly operator-dependent ultrasound scanning, skill assessment approaches evaluate operator competence given available data, such as acquired images and tracked probe movement. Operator skill level can be quantified by the completeness, speed, and precision of performing a clinical task, such as biometry. Such clinical tasks are increasingly becoming assisted or even replaced by automated machine learning models. In addition to measurement, operators need to be competent at the upstream task of acquiring images of sufficient quality. To provide computer assistance for this task requires a new definition of skill. METHODS: This paper focuses on the task of selecting ultrasound frames for biometry, for which operator skill is assessed by quantifying how well the tasks are performed with neural network-based frame classifiers. We first develop a frame classification model for each biometry task, using a novel label-efficient training strategy. Once these task models are trained, we propose a second task model-specific network to predict two skill assessment scores, based on the probability of identifying positive frames and accuracy of model classification. RESULTS: We present comprehensive results to demonstrate the efficacy of both the frame-classification and skill-assessment networks, using clinically acquired data from two biometry tasks for a total of 139 subjects, and compare the proposed skill assessment with metrics of operator experience. CONCLUSION: Task model-specific skill assessment is feasible and can be predicted by the proposed neural networks, which provide objective assessment that is a stronger indicator of task model performance, compared to existing skill assessment methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Wang et al. assessed completeness, precision, and speed of sonographic performance to quantify operator skill level.
“Completeness, precision, and speed of sonographic performance were assessed by Wang et al. to quantify operator skill level [156].”
From Evolving the Era of 5D Ultrasound? A Systematic Literature Review on the Applications for Artificial Intelligence Ultrasound Imaging in Obstetrics and Gynecology (2023) · cites this paper as [156]
The Abstract (Purpose; page unavailable) says operator skill can be quantified by completeness, speed, and precision. But the Method (Skill assessment predictors; page unavailable) describes the measures Wang et al. actually assessed as expected positive task prediction and expected task prediction accuracy, not assessments of all three named dimensions. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Assessment of clinical competence using objective structured examination.
Harden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975) · BMJ · doi:10.1136/bmj.1.5955.447 · Cited by 1,687 · see citing works
Fast Normalized Cross-Correlation
Yoo, J.-C., & Han, T. H. (2009) · Circuits Systems and Signal Processing · doi:10.1007/s00034-009-9130-7 · Cited by 1,128 · see citing works
Practice guidelines for performance of the routine mid-trimester fetal ultrasound scan
Salomon, L. J., Alfirevic, Z., Berghella, V., Bilardo, C., Hernandez‐Andrade, E., Johnsen, S. L., Kalache, K., Leung, K. ‐., Malinger, G., Munoz, H., et al. (2011) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.8831 · Cited by 1,006 · see citing works
SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound
Baumgartner, C. F., Kamnitsas, K., Matthew, J., Fletcher, T. P., Smith, S., Koch, L. M., Kainz, B., & Rueckert, D. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2712367 · Cited by 416 · see citing works
Tolsgaard, M. G., Todsen, T., Sørensen, J. L., Ringsted, C. V., Lorentzen, T., Ottesen, B., & Tabor, A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057687 · Cited by 208 · see citing works
Intra‐ and interobserver variability in fetal ultrasound measurements
Sarris, I., Ioannou, C., Chamberlain, P., Ohuma, E., Roseman, F., Hoch, L., Altman, D. G., Papageorghiou, A. T., & for the International Fetal and Newborn Growth Consortium for the 21 Century (INTERGROWTH‐21) (2012) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.10082 · Cited by 195 · see citing works
Self-citationUltrasound Competency Assessment in Emergency Medicine Residency Programs
Amini, R., Adhikari, S., & Fiorello, A. B. (2014) · Academic Emergency Medicine · doi:10.1111/acem.12408 · Cited by 95 · see citing works
Drukker, L., Sharma, H., Droste, R., Alsharid, M., Chatelain, P., Noble, J. A., & Papageorghiou, A. T. (2021) · Scientific Reports · doi:10.1038/s41598-021-92829-1 · Cited by 59 · see citing works
Self-citationImage quality assessment for machine learning tasks using meta-reinforcement learning
Saeed, S. U., Fu, Y., Stavrinides, V., Baum, Z. M. C., Yang, Q., Rusu, M., Fan, R. E., Sonn, G. A., Noble, J. A., Barratt, D. C., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102427 · Cited by 50 · see citing works
Self-citationAutoFB: Automating Fetal Biometry Estimation from Standard Ultrasound Planes
Bano, S., Dromey, B., Vasconcelos, F., Napolitano, R., David, A. L., Peebles, D., & Stoyanov, D. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87234-2_22 · Cited by 39 · see citing works
Self-Supervised Contrastive Video-Speech Representation Learning for Ultrasound
Jiao, J., Cai, Y., Alsharid, M., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59716-0_51 · Cited by 38 · see citing works
Self-citationZiesmann, M. T., Park, J., Unger, B. J., et al. (2015) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000000813 · Cited by 31 · see citing works
Automated Selection of Standardized Planes from Ultrasound Volume
Rahmatullah, B., Papageorghiou, A. T., & Noble, J. A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24319-6_5 · Cited by 23 · see citing works
Self-citationLee, L. H., Bradburn, E., Papageorghiou, A. T., & Noble, J. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60334-2_2 · Cited by 15 · see citing works
Self-citationSharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2021) · 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi48211.2021.9433863 · Cited by 15 · see citing works
Self-citationLearning Image Quality Assessment by Reinforcing Task Amenable Data Selection
Saeed, S. U., Fu, Y., Baum, Z. M. C., Yang, Q., Rusu, M., Fan, R. E., Sonn, G. A., Barratt, D. C., & Hu, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78191-0_58 · Cited by 15 · see citing works
Self-citationDifferentiating Operator Skill During Routine Fetal Ultrasound Scanning Using Probe Motion Tracking
Wang, Y., Droste, R., Jiao, J., Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60334-2_18 · Cited by 13 · see citing works
Self-citationAdaptable Image Quality Assessment Using Meta-Reinforcement Learning of Task Amenability
Saeed, S. U., Fu, Y., Stavrinides, V., Baum, Z. M. C., Yang, Q., Rusu, M., Fan, R. E., Sonn, G. A., Noble, J. A., Barratt, D. C., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87583-1_19 · Cited by 11 · see citing works
Self-citationLous, M. L., Despinoy, F., Klein, M., et al. (2020) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000465 · Cited by 8 · see citing works
Ultrasound video analysis for skill level assessment in FAST ultrasound
Tyrrell, R. E., & Holden, M. (2020) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2020.1835549 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Jost, E., Kosian, P., Jimenez Cruz, J., Albarqouni, S., Gembruch, U., Strizek, B., & Recker, F. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12216833 · Cited by 44 · see citing works
- Authors
- Wang, Y., Yang, Q., Drukker, L., Papageorghiou, A., Hu, Y., & Noble, J. A.
- Affiliations
- University of Oxford (GB)
- University College London (GB)
- Corresponding author
- Wang, Yipei
- Venue
- International Journal of Computer Assisted Radiology and Surgery
- Published
- 2022-05-12
- Volume
- 17
- Issue
- 8
- Pages
- 1437-1444
- ISSN
- 1861-6410 (ISSN-L)
- 1861-6429
- PMC
- PMC9307537
- OpenAlex
- W4280496832
- Cited by
- 12
- References
- 23
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 2 · 2023: 3 · 2022: 2
- FWCI
- 0.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Journal of Computer Assisted Radiology and Surgery (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- St George's Online Research Archive (St George's University of London) (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- H2020 European Research Council (ERC-ADG-2015 694581)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Ultrasound in Clinical Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- operator skill assessment, prenatal ultrasound, ultrasound biometry, frame classification, operator competence, artificial neural networks
- MeSH terms
- Machine Learning, Female, Humans, Pregnancy, Task Performance and Analysis, Ultrasonography, Prenatal, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
