Automatic Pharyngeal Phase Recognition in Untrimmed Videofluoroscopic Swallowing Study Using Transfer Learning with Deep Convolutional Neural Networks
Open accessThe DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
Background: Video fluoroscopic swallowing study (VFSS) is considered as the gold standard diagnostic tool for evaluating dysphagia. However, it is time consuming and labor intensive for the clinician to manually search the recorded long video image frame by frame to identify the instantaneous swallowing abnormality in VFSS images. Therefore, this study aims to present a deep leaning-based approach using transfer learning with a convolutional neural network (CNN) that automatically annotates pharyngeal phase frames in untrimmed VFSS videos such that frames need not be searched manually. Methods: To determine whether the image frame in the VFSS video is in the pharyngeal phase, a single-frame baseline architecture based the deep CNN framework is used and a transfer learning technique with fine-tuning is applied. Results: Compared with all experimental CNN models, that fine-tuned with two blocks of the VGG-16 (VGG16-FT5) model achieved the highest performance in terms of recognizing the frame of pharyngeal phase, that is, the accuracy of 93.20 (±1.25)%, sensitivity of 84.57 (±5.19)%, specificity of 94.36 (±1.21)%, AUC of 0.8947 (±0.0269) and Kappa of 0.7093 (±0.0488). Conclusions: Using appropriate and fine-tuning techniques and explainable deep learning techniques such as grad CAM, this study shows that the proposed single-frame-baseline-architecture-based deep CNN framework can yield high performances in the full automation of VFSS video analysis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From an algorithmic perspective, these two problems have always been tackled independently, for example: the pharyngeal phase has usually been detected in a video segmentation fashion using convolutional neural networks (CNNs) [9,14,15], whereas the bolus has been localized via image segmentation algorithms [16].”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Our results are in line with Lee et al. [15], where the pharyngeal phase could be detected with accuracy higher than 0.9 using a VGG16-based approach.”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“In particular, the authors demonstrated that retraining all VGG16’s blocks led to better performance”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
In the citing paper
“Recent results obtained by Lee et al. [9,15] demonstrated that this phase can be detected with accuracy higher than 90%.”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
In the citing paper
“More recently, Lee et al. [15] demonstrated that comparable performance in detecting the pharyngeal phase could be obtained with 2DCNNs [39], with a significant reduction of computational complexity compared to 3DCNNs.”
From Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: killing two birds with one stone? (2022) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“This experiment was conducted using six different CNNs with different degrees of fine-tuning using VGG-16 [21] as the base CNN. VGG-16 is a pre-trained CNN developed from the Visual Geometry Group, Department of Engineering Science, University of Oxford.”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 3×
“each frame image from VFSS videos differs from object classification and natural images, they can share similar learned features [24].”
BackgroundA survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · cited 1×
“The VGG architecture has been widely applied and considered as a state-of-the-art architecture in both general and medical fields for various vision tasks, such as image feature extraction, image classification or object detection [22].”
BackgroundCross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881 · cited 2×
“The 258 video clips selected as the training dataset were randomly segmented into five folds to perform five-fold cross validation to evaluate the model training while avoiding overfitting or bias [25].”
MethodLearning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · cited 1×
“In particular, inspired by the recent success of temporal action detection technology on action classification and action recognition in videos, such as three-dimensional convolutional networks (3DConvNets) [14,15], medical researchers have attempted to adopt these techniques to detect the pharyngeal phase in the VFSS…”
BackgroundTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 1×
“All the experiments were performed using the Keras [26] library and TensorFlow [27] backend engine.”
MethodLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 4×
“Although a single-frame-baseline architecture was used in this study, that is, the most basic arc itecture among larg -scale video c ssification technique using the CNN [19], the proposed framework showed the possibility to achieve a substantial level of discriminant ability (AUC = 0.”
MethodInfluentialHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“When the training dataset is relatively small, transferring a CNN pretrained by a large annotated dataset and fine-tuning it for a specific task can be an efficient method for achieving acceptable goals and lower training costs [23].”
MethodConvolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · cited 1×
“However, based on a previous study that described the effects and mechanisms of fine-tuning on deep CNNs, when setting certain convolutional layers of a deep CNN model be trainable, the CNN model can be further specialized for specific classifying tasks [24,31].”
Background
Research works printed but not linked: 1
- Horiguchi, 2011, Screening tests in evaluating swallowing function, JMAJ, 54, 31
Cited by
2023 IEEE International Conference on Image Processing (ICIP)
(2023) · doi:10.1109/icip49359.2023 · Cited by 353
Kim, H.-I., Kim, Y., Kim, B., Shin, D. Y., Lee, S. J., & Choi, S. (2021) · Diagnostics · doi:10.3390/diagnostics11071147 · Cited by 35
Jeong, S. Y., Kim, J. M., Park, J. E., Baek, S. J., & Yang, S. N. (2023) · Scientific Reports · doi:10.1038/s41598-023-44802-3 · Cited by 31
“We compared our model with those of prior studies on VFSS. Lee et al. proposed a model based on VGG-16, a neural network with 16 weight layers that is widely used for image recognition 24,33 .”
MethodResultA New Smart Home Intruder Detection System Based on Deep Learning
Ali, H. H., Naif, J. R., & Humood, W. R. (2023) · Al-Mustansiriyah Journal of Sciences · doi:10.23851/mjs.v34i2.1267 · Cited by 34
“Pre-trained CNNs are applied as the backbone of most state-of-the-art computer vision (CV) models [28, 29].”
MethodExplainability of deep learning models in medical video analysis: a survey
Kolárik, M., Sarnovský, M., Paralič, J., & Babič, F. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1253 · Cited by 31
Bandini, A., Smaoui, S., & Steele, C. M. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107058 · Cited by 16
“[14], where the pharyngeal phase could be detected with accuracy higher than 0.”
BackgroundMethodInfluentialPark, D., Kim, Y., Kang, H., Lee, J., Choi, J. R., Kim, T., Lee, S.-E., Son, S., Kim, M., & Kim, I. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108241 · Cited by 16
“Two studies reported an attempt to detect the pharyngeal swallowing reflex time by presenting a DL model that recognizes a specific stage (pharyngeal phase) in VFSS images, which can enable quantitative evaluation of findings such as swallowing reflex delay [22, 21].”
BackgroundShu, K., Mao, S., Zhang, Z., Coyle, J. L., & Sejdić, E. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108505 · Cited by 16
Artificial Intelligence in Videofluoroscopy Swallow Study Analysis: A Comprehensive Review
Sanjeevi, G., Gopalakrishnan, U., Pathinarupothi, R. K., & Iyer, K. S. (2025) · Dysphagia · doi:10.1007/s00455-025-10812-8 · Cited by 12
“Expanding on this, Lee et al. [22] employed a transfer learning approach classifying the VFSS frames into pharyngeal and non-pharyngeal frames (see Fig.”
Video-SwinUNet: Spatio-temporal Deep Learning Framework for VFSS Instance Segmentation
Zeng, C., Yang, X., Smithard, D. G., Mirmehdi, M., Gambaruto, A. M., & Burghardt, T. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222212 · Cited by 14
Machine learning in the evaluation of voice and swallowing in the head and neck cancer patient
Srinivasan, Y., Liu, A. Y., & Rameau, A. (2023) · Current Opinion in Otolaryngology & Head & Neck Surgery · doi:10.1097/moo.0000000000000948 · Cited by 10
Bandini, A., & Steele, C. M. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9629562 · Cited by 12
Artificial intelligence in the diagnosis and management of dysphagia: a scoping review
Silva, R. D. D., Almeida, S. B., Gonçalves, F. M., Zeigelboim, B. S., Stechman-Neto, J., Schroder, A. G. D., Nascimento, W. V., Santos, R. S., & Araujo, C. M. D. (2025) · CoDAS · doi:10.1590/2317-1782/e20240305en · Cited by 6
“Sound resources as an auxiliary method in evaluation were also used (17,21,25,26,28,34-46) .”
Fakhry, A., Antony, S. M., Park, E., & Lee, J. T. (2025) · IEEE Access · doi:10.1109/access.2025.3573282 · Cited by 8
Sreedevi, E. V., Iyer, S., Thankappan, K., Janakiram, C., Karuveettil, V., Krishnan, R., Guntha, R., Roe, J. W. G., & Menon, J. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-108726 · Cited by 2
Bandini, A., Lasala, A., Péladeau-Pigeon, M., Dharmarathna, I., Gandhi, P., Meng, D., Milne, H., Panes, V., Simmons, M. M., & Steele, C. M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.111361 · Cited by 3
Rapid Detection of Penetration-Aspiration from Fluoroscopic Videos in Dysphagia Patients
Sanjeevi, G., Gopalakrishnan, U., Pathinarupothi, R. K., Arya, C. J., & Subramania Iyer, K. (2023) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe60311.2023.00073 · Cited by 2
“Additionally, Lee et al. [4] also proposed a deep learning model using transfer learning to automatically detect pharyngeal phase frames from the VFSS.”
MethodVideo-SwinUNet: Spatio-temporal Deep Learning Framework for VFSS Instance SegmentationPreprint
Zeng, C., Yang, X., Smithard, D. G., Mirmehdi, M., Gambaruto, A. M., & Burghardt, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11325 · Cited by 6
Deep Learning-Based Classification of Dysphagia Severity Using M-Mode Ultrasound ImagingPreprint
Choi, H., Sung, J. H., Kim, H., Kim, S., Jeong, T. T., Lee, J. J., Shin, S.-H., & Kim, C. (2025) · Research Square · doi:10.21203/rs.3.rs-7823321/v1 · Cited by 1
da Rocha, M. M. L., van der Molen, L., Neijman, M., van Alphen, M. J. A., van den Brekel, M. W. M., & Siepel, F. J. (2025) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-025-03556-1 · Cited by 1
Metadata
- Authors
- Lee, K.-S., Lee, E., Choi, B., & Pyun, S.-B.
- Affiliations
- Korea University (KR)
- Corresponding author
- Lee, Ki-Sun
- Pyun, Sung-Bom
- Venue
- Diagnostics
- Publisher
- MDPI AG
- Published
- 2021-02-13
- Volume
- 11
- Issue
- 2
- Pages
- 300
- ISSN
- 2075-4418 (ISSN-L)
- PMC
- PMC7918932
- OpenAlex
- W3133477037
- Cited by
- 25
- References
- 38
- Citations by year
- 2025: 7 · 2024: 2 · 2023: 13 · 2022: 1 · 2021: 2
- FWCI
- 5.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Diagnostics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2019R1I1A1A01062961)
- Korea University Grant (K2008481)
- Topics
- Health Sciences › Health Professions › Speech and Hearing › Dysphagia Assessment and Management
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- videofluoroscopic swallowing study, dysphagia, automatic video annotation, transfer learning, convolutional neural networks, Grad-CAM
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.