Automatic Pharyngeal Phase Recognition in Untrimmed Videofluoroscopic Swallowing Study Using Transfer Learning with Deep Convolutional Neural Networks
Open accessThe DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
Background: Video fluoroscopic swallowing study (VFSS) is considered as the gold standard diagnostic tool for evaluating dysphagia. However, it is time consuming and labor intensive for the clinician to manually search the recorded long video image frame by frame to identify the instantaneous swallowing abnormality in VFSS images. Therefore, this study aims to present a deep leaning-based approach using transfer learning with a convolutional neural network (CNN) that automatically annotates pharyngeal phase frames in untrimmed VFSS videos such that frames need not be searched manually. Methods: To determine whether the image frame in the VFSS video is in the pharyngeal phase, a single-frame baseline architecture based the deep CNN framework is used and a transfer learning technique with fine-tuning is applied. Results: Compared with all experimental CNN models, that fine-tuned with two blocks of the VGG-16 (VGG16-FT5) model achieved the highest performance in terms of recognizing the frame of pharyngeal phase, that is, the accuracy of 93.20 (±1.25)%, sensitivity of 84.57 (±5.19)%, specificity of 94.36 (±1.21)%, AUC of 0.8947 (±0.0269) and Kappa of 0.7093 (±0.0488). Conclusions: Using appropriate and fine-tuning techniques and explainable deep learning techniques such as grad CAM, this study shows that the proposed single-frame-baseline-architecture-based deep CNN framework can yield high performances in the full automation of VFSS video analysis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From an algorithmic perspective, these two problems have always been tackled independently, for example: the pharyngeal phase has usually been detected in a video segmentation fashion using convolutional neural networks (CNNs) [9,14,15], whereas the bolus has been localized via image segmentation algorithms [16].”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In particular, the authors demonstrated that retraining all VGG16’s blocks led to better performance”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
In the citing paper
“Recent results obtained by Lee et al. [9,15] demonstrated that this phase can be detected with accuracy higher than 90%.”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
In the citing paper
“Our results are in line with Lee et al. [15], where the pharyngeal phase could be detected with accuracy higher than 0.9 using a VGG16-based approach.”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
In the citing paper
“More recently, Lee et al. [15] demonstrated that comparable performance in detecting the pharyngeal phase could be obtained with 2DCNNs [39], with a significant reduction of computational complexity compared to 3DCNNs.”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“This experiment was conducted using six different CNNs with different degrees of fine-tuning using VGG-16 [21] as the base CNN. VGG-16 is a pre-trained CNN developed from the Visual Geometry Group, Department of Engineering Science, University of Oxford.”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 3×
“each frame image from VFSS videos differs from object classification and natural images, they can share similar learned features [24].”
BackgroundA survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · cited 1×
“The VGG architecture has been widely applied and considered as a state-of-the-art architecture in both general and medical fields for various vision tasks, such as image feature extraction, image classification or object detection [22].”
BackgroundCross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 2×
“The 258 video clips selected as the training dataset were randomly segmented into five folds to perform five-fold cross validation to evaluate the model training while avoiding overfitting or bias [25].”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“In particular, inspired by the recent success of temporal action detection technology on action classification and action recognition in videos, such as three-dimensional convolutional networks (3DConvNets) [14,15], medical researchers have attempted to adopt these techniques to detect the pharyngeal phase in the VFSS…”
BackgroundTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 1×
“All the experiments were performed using the Keras [26] library and TensorFlow [27] backend engine.”
MethodLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 4×
“Although a single-frame-baseline architecture was used in this study, that is, the most basic arc itecture among larg -scale video c ssification technique using the CNN [19], the proposed framework showed the possibility to achieve a substantial level of discriminant ability (AUC = 0.”
MethodInfluentialHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“When the training dataset is relatively small, transferring a CNN pretrained by a large annotated dataset and fine-tuning it for a specific task can be an efficient method for achieving acceptable goals and lower training costs [23].”
MethodConvolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · cited 1×
“However, based on a previous study that described the effects and mechanisms of fine-tuning on deep CNNs, when setting certain convolutional layers of a deep CNN model be trainable, the CNN model can be further specialized for specific classifying tasks [24,31].”
BackgroundLarge-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,387
II. MORE THAN JUST CONVENIENT: THE SCIENTIFIC MERITS OF HOMOGENEOUS CONVENIENCE SAMPLES
Jager, J., Putnick, D. L., & Bornstein, M. H. (2017) · Monographs of the Society for Research in Child Development · doi:10.1111/mono.12296 · Cited by 1,058
Keras: The Python Deep Learning library
Chollet, F. (2018) · Astrophysics Source Code Library · Cited by 1,002
Gordon, C., Hewer, R. L., & Wade, D. T. (1987) · BMJ · doi:10.1136/bmj.295.6595.411 · Cited by 651
Ertekin, C., & Aydogdu, I. (2003) · Clinical Neurophysiology · doi:10.1016/s1388-2457(03)00237-2 · Cited by 536 · cited 1×
“Clinically, the analysis of the pharyngeal phase in the VFSS video is important to the entire swallowing process, as it can identify any abnormalities that can result in serious medical complications, such as aspiration pneumonia or asphyxia [30].”
BackgroundAGA technical review on management of oropharyngeal dysphagia
Cook, I. J., & Kahrilas, P. J. (1999) · Gastroenterology · doi:10.1016/s0016-5085(99)70144-7 · Cited by 505 · cited 1×
“Videofluoroscopy swallowing study (VFSS) or a modified barium swallow study, is considered the gold standard tool for studying the oral and pharyngeal processes for evaluating the swallowing process of dysphasia patients [8].”
MethodValueva, M. V., Nagornov, N. N., Lyakhov, P. A., Valuev, G. V., & Chervyakov, N. I. (2020) · Mathematics and Computers in Simulation · doi:10.1016/j.matcom.2020.04.031 · Cited by 478 · cited 1×
“In deep learning, the CNN (or ConvNet) is a class of deep neural network that is the most typically applied in analyzing visual images [20].”
BackgroundICU-Acquired Swallowing Disorders
Macht, M., Wimbish, T., Bodine, C., & Moss, M. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31829caf33 · Cited by 261 · cited 1×
“This not only shortens the reestablishment of the overall health status but also reduces the overall rehabilitation efforts and costs [6,7].”
BackgroundExploring Large-scale Public Medical Image Datasets
Oakden-Rayner, L. (2020) · Academic Radiology · doi:10.1016/j.acra.2019.10.006 · Cited by 250 · cited 1×
“It is assumed that for images of unlabeled subsets other than the pharyngeal phase, the deep CNN algorithm generated hidden stratification [36].”
BackgroundLearning Spatiotemporal Features Using 3DCNN and Convolutional LSTM for Gesture Recognition
Zhang, L., Zhu, G., Shen, P., Song, J., Shah, S. A., & Bennamoun, M. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.369 · Cited by 241 · cited 1×
“In particular, inspired by the recent success of temporal action detection technology on action classification and action recognition in videos, such as three-dimensional convolutional networks (3DConvNets) [14,15], medical researchers have attempted to adopt these techniques to detect the pharyngeal phase in the VFSS…”
BackgroundScreening for Oropharyngeal Dysphagia in Stroke: Insufficient Evidence for Guidelines
Martino, R., Pron, G., & Diamant, N. (2000) · Dysphagia · doi:10.1007/s004559910006 · Cited by 241 · cited 1×
“This not only shortens the reestablishment of the overall health status but also reduces the overall rehabilitation efforts and costs [6,7].”
BackgroundA protocol for the videofluorographic swallowing study
Palmer, J. B., Kuhlemeier, K. V., Tippett, D. C., & Lynch, C. (1993) · Dysphagia · doi:10.1007/bf01354540 · Cited by 206 · cited 3×
“Clinicians repeatedly analyze the recorded video to evaluate abnormalities associated with the swallowing process [9].”
BackgroundMethodInfluentialLee, K.-S., Jung, S.-K., Ryu, J.-J., Shin, S.-W., & Choi, J. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9020392 · Cited by 187
Self-citationComparison of Transferred Deep Neural Networks in Ultrasonic Breast Masses Discrimination
Xiao, T., Liu, L., Li, K., Qin, W., Yu, S., & Li, Z. (2018) · BioMed Research International · doi:10.1155/2018/4605191 · Cited by 168 · cited 1×
“Although the target medical image and the analysis purpose are different, the results are similar to those of previous studies [32–34] using the transfer learning of a deep CNN via fine-tuning.”
ResultOropharyngeal Swallow Efficiency as a Representative Measure of Swallowing Function
Rademaker, A. W., Pauloski, B. R., Logemann, J. A., & Shanahan, T. K. (1994) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jshr.3702.314 · Cited by 167 · cited 1×
“According to medical criteria [28,29], the beginning of the pharyngeal phase is defined as the moment when the head of the bolus is propelled to the pharynx, when the soft palate elevates and presses against the posterior wall of the pharynx.”
BackgroundA Pursuit of Temporal Accuracy in General Activity DetectionPreprint
Xiong, Y., Zhao, Y., Wang, L., Lin, D., & Tang, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02716 · Cited by 147 · cited 3×
“17 s) [17].”
BackgroundInter- and Intrajudge Reliability for Videofluoroscopic Swallowing Evaluation Measures
McCullough, G. H., Wertz, R. T., Rosenbek, J. C., Mills, R. H., Webb, W. G., & Ross, K. (2001) · Dysphagia · doi:10.1007/pl00021291 · Cited by 144 · cited 2×
“Furthermore, another study reported that the consistency of the VFSS cannot be guaranteed owing to the subjectivity of the examiner when performing frame-by-frame analysis [12].”
BackgroundAssessment and Early Diagnosis of Dysphagia
Wieseke, A., Bantz, D. L., Siktberg, L. L., & Dillard, N. (2008) · Geriatric Nursing · doi:10.1016/j.gerinurse.2007.12.001 · Cited by 92 · cited 1×
“Dysphagia is defined as a clinical symptom of difficulty swallowing foods [1].”
BackgroundLee, K.-S., Kim, J., Jeon, E., Choi, W. S., Kim, N. H., & Lee, K. Y. (2020) · Journal of Personalized Medicine · doi:10.3390/jpm10040213 · Cited by 74 · cited 1×
“Although the target medical image and the analysis purpose are different, the results are similar to those of previous studies [32–34] using the transfer learning of a deep CNN via fine-tuning.”
ResultSelf-citationAutomatic hyoid bone detection in fluoroscopic images using deep learning
Zhang, Z., Coyle, J. L., & Sejdić, E. (2018) · Scientific Reports · doi:10.1038/s41598-018-30182-6 · Cited by 70 · cited 2×
“A previous study reported that VFSS analysis is time consuming and laborious to a clinician [11].”
BackgroundClinical and videofluoroscopic evaluation of swallowing disorders.
Ott, D. J., & Pikna, L. A. (1993) · American Journal of Roentgenology · doi:10.2214/ajr.161.3.8352094 · Cited by 65 · cited 1×
“According to medical criteria [28,29], the beginning of the pharyngeal phase is defined as the moment when the head of the bolus is propelled to the pharynx, when the soft palate elevates and presses against the posterior wall of the pharynx.”
BackgroundMoraes, D. P., Sassi, F. C., Mangilli, L. D., Zilberstein, B., & de Andrade, C. R. F. (2013) · Critical Care · doi:10.1186/cc13069 · Cited by 58 · cited 1×
“Hence, underlying health conditions may interact with dysphagia to produce aspiration, pneumonia, and/or respiratory compromise [4].”
BackgroundLee, J. T., Park, E., Hwang, J.-M., Jung, T.-D., & Park, D. (2020) · Scientific Reports · doi:10.1038/s41598-020-71713-4 · Cited by 49 · cited 3×
“Bec use the pharyngeal phase in the entire long VFSS videos occurs during the short frame sequence, a recent study reported tha this cutting- dg deep learning technology may present limitati ns in recognizing activity during short frames in long-sequence videos [17,18].”
BackgroundResultShem, K. L., Castillo, K., Wong, S., Chang, J., Kao, M. J., & Kolakowsky‐Hayner, S. A. (2012) · PM&R · doi:10.1016/j.pmrj.2012.01.002 · Cited by 45 · cited 2×
“VFSS can illustrate the physiological process of the entire s allo ing activity, including the motions of the jaws, tongue, palate, pharynx, larynx, esophagus and bolus of food [10].”
BackgroundHan, H., Shin, G., Jun, A.-Y., Park, T., Ko, D.-H., Choi, E., & Kim, Y. (2016) · Annals of Rehabilitation Medicine · doi:10.5535/arm.2016.40.1.88 · Cited by 43 · cited 2×
“In particular, the recognition of the pharyngeal phase frames in VFSS by clinicians is crucial for shorte ing the examination time and revealing abnormalities in swallowing because aspiration or penetration occurs during the pharyngeal phase [13].”
BackgroundSwallowing Complications After Endotracheal Extubation
Heffner, J. E. (2010) · CHEST Journal · doi:10.1378/chest.09-2477 · Cited by 41 · cited 1×
“Moreover, dysphagia may interfere with nutrition, delay clinical recovery and even results in death if not diagnosed early and appropriately [5].”
BackgroundLee, J. T., Park, E., & Jung, T.-D. (2019) · Sensors · doi:10.3390/s19183873 · Cited by 30
Research works printed but not linked: 1
- Horiguchi, 2011, Screening tests in evaluating swallowing function, JMAJ, 54, 31
Cited by
2023 IEEE International Conference on Image Processing (ICIP)
(2023) · doi:10.1109/icip49359.2023 · Cited by 353
Kim, H.-I., Kim, Y., Kim, B., Shin, D. Y., Lee, S. J., & Choi, S. (2021) · Diagnostics · doi:10.3390/diagnostics11071147 · Cited by 35
Jeong, S. Y., Kim, J. M., Park, J. E., Baek, S. J., & Yang, S. N. (2023) · Scientific Reports · doi:10.1038/s41598-023-44802-3 · Cited by 31
“We compared our model with those of prior studies on VFSS. Lee et al. proposed a model based on VGG-16, a neural network with 16 weight layers that is widely used for image recognition 24,33 .”
MethodResultA New Smart Home Intruder Detection System Based on Deep Learning
Ali, H. H., Naif, J. R., & Humood, W. R. (2023) · Al-Mustansiriyah Journal of Sciences · doi:10.23851/mjs.v34i2.1267 · Cited by 34
“Pre-trained CNNs are applied as the backbone of most state-of-the-art computer vision (CV) models [28, 29].”
MethodExplainability of deep learning models in medical video analysis: a survey
Kolárik, M., Sarnovský, M., Paralič, J., & Babič, F. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1253 · Cited by 31
Metadata
- Authors
- Lee, K.-S., Lee, E., Choi, B., & Pyun, S.-B.
- Affiliations
- Korea University (KR)
- Corresponding author
- Lee, Ki-Sun
- Pyun, Sung-Bom
- Venue
- Diagnostics
- Publisher
- MDPI AG
- Published
- 2021-02-13
- Volume
- 11
- Issue
- 2
- Pages
- 300
- ISSN
- 2075-4418 (ISSN-L)
- PMC
- PMC7918932
- OpenAlex
- W3133477037
- Cited by
- 25
- References
- 38
- Citations by year
- 2025: 7 · 2024: 2 · 2023: 13 · 2022: 1 · 2021: 2
- FWCI
- 5.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Diagnostics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2019R1I1A1A01062961)
- Korea University Grant (K2008481)
- Topics
- Health Sciences › Health Professions › Speech and Hearing › Dysphagia Assessment and Management
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- videofluoroscopic swallowing study, dysphagia, automatic video annotation, transfer learning, convolutional neural networks, Grad-CAM
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.