CoChat

Automatic Pharyngeal Phase Recognition in Untrimmed Videofluoroscopic Swallowing Study Using Transfer Learning with Deep Convolutional Neural Networks

Open access

The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.

Abstract

Background: Video fluoroscopic swallowing study (VFSS) is considered as the gold standard diagnostic tool for evaluating dysphagia. However, it is time consuming and labor intensive for the clinician to manually search the recorded long video image frame by frame to identify the instantaneous swallowing abnormality in VFSS images. Therefore, this study aims to present a deep leaning-based approach using transfer learning with a convolutional neural network (CNN) that automatically annotates pharyngeal phase frames in untrimmed VFSS videos such that frames need not be searched manually. Methods: To determine whether the image frame in the VFSS video is in the pharyngeal phase, a single-frame baseline architecture based the deep CNN framework is used and a transfer learning technique with fine-tuning is applied. Results: Compared with all experimental CNN models, that fine-tuned with two blocks of the VGG-16 (VGG16-FT5) model achieved the highest performance in terms of recognizing the frame of pharyngeal phase, that is, the accuracy of 93.20 (±1.25)%, sensitivity of 84.57 (±5.19)%, specificity of 94.36 (±1.21)%, AUC of 0.8947 (±0.0269) and Kappa of 0.7093 (±0.0488). Conclusions: Using appropriate and fine-tuning techniques and explainable deep learning techniques such as grad CAM, this study shows that the proposed single-frame-baseline-architecture-based deep CNN framework can yield high performances in the full automation of VFSS video analysis.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/diagnostics11020300

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 (A) Structural anatomy of normal swallowing of thick liquid bolus in (B) oral phase, (C–E) pharyngeal phase and (F) esophageal phase.
Figure 1 (A) Structural anatomy of normal swallowing of thick liquid bolus in (B) oral phase, (C–E) pharyngeal phase and (F) esophageal phase.CC BY · PubMed Central
Figure 2 Conceptual diagram of framework proposed herein.
Figure 2 Conceptual diagram of framework proposed herein.CC BY · PubMed Central
Figure 3 Schematic diagram of six experimental groups based on fine-tuning degree in VGG-16 backbone convolutional neural network (CNN).
Figure 3 Schematic diagram of six experimental groups based on fine-tuning degree in VGG-16 backbone convolutional neural network (CNN).CC BY · PubMed Central
Figure 4 Overview of five-fold cross validation applied in this study.
Figure 4 Overview of five-fold cross validation applied in this study.CC BY · PubMed Central
Figure 5 Conceptual diagram of pharyngeal phase recognitions results in test video fluoroscopic swallowing study (VFSS) video clip. Blue boxes mean pharyngeal phase frames labelled by specialists in VFSS video clips. Red boxes mean pharyngeal phase frames predicted by deep CNN model in VFSS video clips. TN: true negative; FP: false positive; TP: true positive; FN: false negative.
Figure 5 Conceptual diagram of pharyngeal phase recognitions results in test video fluoroscopic swallowing study (VFSS) video clip. Blue boxes mean pharyngeal phase frames labelled by specialists in VFSS video clips. Red boxes mean pharyngeal phase frames predicted by deep CNN model in VFSS video clips. TN: true negative; FP: false positive; TP: true positive; FN: false negative.CC BY · PubMed Central
Figure 6 The number of trainable parameters (Trainable), the number of non-trainable parameters (Non-trainable) and the total training time (Time) of each experimental group in this study.
Figure 6 The number of trainable parameters (Trainable), the number of non-trainable parameters (Non-trainable) and the total training time (Time) of each experimental group in this study.CC BY · PubMed Central
Figure 7 The changes of some indexes for model reliability (area under the receiver operating characteristic curve (AUC), Matthews correlation coefficient (MCC) and Kappa) and prediction scores (false positive (FPR), false negative (FNR), positive prediction (PPV) and negative prediction (NPV)) according to the numbers of trainable blocks in the deep CNN (VGG-16).
Figure 7 The changes of some indexes for model reliability (area under the receiver operating characteristic curve (AUC), Matthews correlation coefficient (MCC) and Kappa) and prediction scores (false positive (FPR), false negative (FNR), positive prediction (PPV) and negative prediction (NPV)) according to the numbers of trainable blocks in the deep CNN (VGG-16).CC BY · PubMed Central
Figure 8 Confusion matrix of best performing classification model (fifth iteration of VGG16-FT5) in this study.
Figure 8 Confusion matrix of best performing classification model (fifth iteration of VGG16-FT5) in this study.CC BY · PubMed Central
Figure 9 Sample frame images of original and gradient-weighted class activation mapping (Grad-CAM) correctly predicted by the best performing classification model (VGG16-FT5) in this study.
Figure 9 Sample frame images of original and gradient-weighted class activation mapping (Grad-CAM) correctly predicted by the best performing classification model (VGG16-FT5) in this study.CC BY · PubMed Central
Figure 10 Sample images of original and Grad-CAM presumed to be falsely positively predicted (A) and falsely negatively predicted (B) by the best performing classification model (VGG16-FT5).
Figure 10 Sample images of original and Grad-CAM presumed to be falsely positively predicted (A) and falsely negatively predicted (B) by the best performing classification model (VGG16-FT5).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Very Deep Convolutional Networks for Large-Scale Image Recognition

    Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×

    “This experiment was conducted using six different CNNs with different degrees of fine-tuning using VGG-16 [21] as the base CNN. VGG-16 is a pre-trained CNN developed from the Visual Geometry Group, Department of Engineering Science, University of Oxford.”
    Method
  • Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

    Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513

  • A survey on transfer learning

    Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 3×

    “each frame image from VFSS videos differs from object classification and natural images, they can share similar learned features [24].”
    Background
  • A survey on deep learning in medical image analysis

    Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · cited 1×

    “The VGG architecture has been widely applied and considered as a state-of-the-art architecture in both general and medical fields for various vision tasks, such as image feature extraction, image classification or object detection [22].”
    Background
  • Cross validatory choice and assessment of statistical predictions

    Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 2×

    “The 258 video clips selected as the training dataset were randomly segmented into five folds to perform five-fold cross validation to evaluate the model training while avoiding overfitting or bias [25].”
    Method
  • Learning Spatiotemporal Features with 3D Convolutional Networks

    Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×

    “In particular, inspired by the recent success of temporal action detection technology on action classification and action recognition in videos, such as three-dimensional convolutional networks (3DConvNets) [14,15], medical researchers have attempted to adopt these techniques to detect the pharyngeal phase in the VFSS…”
    Background
  • TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 1×

    “All the experiments were performed using the Keras [26] library and TensorFlow [27] backend engine.”
    Method
  • Large-Scale Video Classification with Convolutional Neural Networks

    Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 4×

    “Although a single-frame-baseline architecture was used in this study, that is, the most basic arc itecture among larg -scale video c ssification technique using the CNN [19], the proposed framework showed the possibility to achieve a substantial level of discriminant ability (AUC = 0.”
    MethodInfluential
  • How transferable are features in deep neural networks?

    Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×

    “When the training dataset is relatively small, transferring a CNN pretrained by a large annotated dataset and fine-tuning it for a specific task can be an efficient method for achieving acceptable goals and lower training costs [23].”
    Method
  • Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

    Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · cited 1×

    “However, based on a previous study that described the effects and mechanisms of fine-tuning on deep CNNs, when setting certain convolutional layers of a deep CNN model be trainable, the CNN model can be further specialized for specific classifying tasks [24,31].”
    Background

Show all 37 references

Research works printed but not linked: 1

  1. Horiguchi, 2011, Screening tests in evaluating swallowing function, JMAJ, 54, 31

Cited by

Metadata

Authors
Lee, K.-S., Lee, E., Choi, B., & Pyun, S.-B.
Affiliations
Korea University (KR)
Corresponding author
  • Lee, Ki-Sun
  • Pyun, Sung-Bom
Venue
Diagnostics
Publisher
MDPI AG
Published
2021-02-13
Volume
11
Issue
2
Pages
300
ISSN
2075-4418 (ISSN-L)
OpenAlex
W3133477037
Cited by
25
References
38
Citations by year
2025: 7 · 2024: 2 · 2023: 13 · 2022: 1 · 2021: 2
FWCI
5.59
Type
journal article
Language
en
Open access
Locations
  • Diagnostics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2019R1I1A1A01062961)
  • Korea University Grant (K2008481)
Topics
  • Health Sciences › Health Professions › Speech and Hearing › Dysphagia Assessment and Management
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Keywords
videofluoroscopic swallowing study, dysphagia, automatic video annotation, transfer learning, convolutional neural networks, Grad-CAM

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.