Deep Learning in Diagnosis of Maxillary Sinusitis Using Conventional Radiography
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Objectives: The aim of this study was to compare the diagnostic performance of a deep learning algorithm with that of radiologists in diagnosing maxillary sinusitis on Waters’ view radiographs. Materials and Methods: Among 80,475 Waters’ view radiographs, examined between May 2003 and February 2017, 9000 randomly selected cases were classified as normal or maxillary sinusitis based on radiographic findings and divided into training (n = 8000) and validation (n = 1000) sets to develop a deep learning algorithm. Two test sets composed of Waters’ view radiographs with concurrent paranasal sinus computed tomography were labeled based on computed tomography findings: one with temporal separation (n = 140) and the other with geographic separation (n = 200) from the training set. Area under the receiver operating characteristics curve (AUC), sensitivity, and specificity of the algorithm and 5 radiologists were assessed. Interobserver agreement between the algorithm and majority decision of the radiologists was measured. The correlation coefficient between the predicted probability of the algorithm and average confidence level of the radiologists was determined. Results: The AUCs of the deep learning algorithm were 0.93 and 0.88 for the temporal and geographic external test sets, respectively. The AUCs of the radiologists were 0.83 to 0.89 for the temporal and 0.75 to 0.84 for the geographic external test sets. The deep learning algorithm showed statistically significantly higher AUC than radiologist in both test sets. In terms of sensitivity and specificity, the deep learning algorithm was comparable to the radiologists. A strong interobserver agreement was noted between the algorithm and radiologists (Cohen κ coefficient, 0.82). The correlation coefficient between the predicted probability of the algorithm and confidence level of radiologists was 0.89 and 0.84 for the 2 test sets, respectively. Conclusions: The deep learning algorithm could diagnose maxillary sinusitis on Waters’ view radiograph with superior AUC and comparable sensitivity and specificity to those of radiologists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
European Position Paper on Rhinosinusitis and Nasal Polyps 2020
Fokkens, W. J., Lund, V. J., Hopkins, C., Hellings, P. W., Kern, R., Reitsma, S., Toppila-Salmi, S., Bernal-Sprekelsen, M., Mullol, J., Alobid, I., et al. (2020) · Rhinology journal · doi:10.4193/rhin20.600 · Cited by 5,882
Machine learning for medical imaging
Erickson, B. J., Korfiatis, P., Akkus, Z., & Kline, T. L. (2017) · RadioGraphics · doi:10.1148/rg.2017160130 · Cited by 1,672
Clinical Practice Guideline (Update): Adult Sinusitis
Rosenfeld, R. M., Piccirillo, J. F., Chandrasekhar, S. S., Brook, I., Ashok Kumar, K., Kramper, M., Orlandi, R. R., Palmer, J. N., Patel, Z. M., Peters, A., et al. (2015) · Otolaryngology–Head and Neck Surgery · doi:10.1177/0194599815572097 · Cited by 1,159
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Computed tomographic study of the common cold
Gwaltney, J. M., Phillips, C. D., Miller, R. D., & Riker, D. K. (1994) · New England Journal of Medicine · doi:10.1056/nejm199401063300105 · Cited by 668
Larson, D. B., Chen, M. C., Lungren, M. P., Halabi, S. S., Stence, N. V., & Langlotz, C. P. (2018) · Radiology · doi:10.1148/radiol.2017170236 · Cited by 456
Becker, A. S., Marcon, M., Ghafoor, S., Wurnig, M. C., Frauenfelder, T., & Boss, A. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000358 · Cited by 352
Research works printed but not linked: 28
- 2017, Implementing machine learning in radiology practice and research, Am J Roentgenol, 208, 754, 10.2214/AJR.16.17224 · DOI · Paper page
- 2017, Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs, Invest Radiol, 52, 281, 10.1097/RLI.0000000000000341 · DOI · Paper page
- 2018, Texture analysis and machine learning for detecting myocardial infarction in noncontrast low-dose computed tomography: unveiling the invisible, Invest Radiol, 53, 338, 10.1097/RLI.0000000000000448 · DOI · Paper page
- 2017, Intravoxel incoherent motion: model-free determination of tissue type in abdominal organs using machine learning, Invest Radiol, 52, 747, 10.1097/RLI.0000000000000400 · DOI
- 1998, Acute sinusitis: a cost-effective approach to diagnosis and treatment, Am Fam Physician, 58, 1795
- 2008, The diagnosis and management of acute and chronic sinusitis, Prim Care, 35, 11, 10.1016/j.pop.2007.09.002 · DOI
- 2011, Rhinosinusitis diagnosis and management for the clinician: a synopsis of recent consensus guidelines, Mayo Clin Proc, 86, 427, 10.4065/mcp.2010.0392 · DOI · Paper page
- 2005, The diagnosis and management of sinusitis: a practice parameter update, J Allergy Clin Immunol, 116, S13, 10.1016/j.jaci.2005.09.048 · DOI · Paper page
- 1991, Paranasal sinuses on MR images of the brain: significance of mucosal thickening, Am J Roentgenol, 156, 381, 10.2214/ajr.156.2.1898819 · doi:10.2214/ajr.156.2.1898819
- 1988, MR imaging of normal nasal cycle: comparison with sinus pathology, J Comput Assist Tomogr, 12, 1014, 10.1097/00004728-198811000-00019 · DOI
- 2009, Reappraising the role of radiography in the diagnosis of chronic rhinosinusitis, Rhinology, 47, 271, 10.4193/Rhin08.147 · doi:10.4193/rhin08.147
- 2005, Study on the reproducibility of the Waters' views of the maxillary sinuses, Rhinology, 43, 55
- 2000, The value of the occipitomental (Waters') view in diagnosis of sinusitis: a comparative study with computed tomography, Clin Radiol, 55, 856, 10.1053/crad.2000.0550 · doi:10.1053/crad.2000.0550
- 1998, Antibiotic treatment of patients with mucosal thickening in the paranasal sinuses, and validation of cut-off points in sinus CT, Rhinology, 36, 7
- 1991, The impact of screening sinus CT on the planning of functional endoscopic sinus surgery, Otolaryngol Head Neck Surg, 105, 802, 10.1177/019459989110500606 · doi:10.1177/019459989110500606
- 1996, Use of symptoms, signs, and blood tests to diagnose acute sinus infections in primary care: comparison with computed tomography, Fam Med, 28, 183
- 2017, ACR Appropriateness Criteria® Sinonasal Disease, J Am Coll Radiol, 14, S550, 10.1016/j.jacr.2017.08.041 · DOI
- 2003, Conventional sinus radiography compared with CT in the diagnosis of acute sinusitis, Dentomaxillofac Radiol, 32, 60, 10.1259/dmfr/65139094 · doi:10.1259/dmfr/65139094
- 1999, Diagnosis and treatment of acute bacterial rhinosinusitis, Evid Rep Technol Assess (Summ), 1
- 2015, Development of the maxillary sinus from birth to age 18. Postnatal growth pattern, Int J Pediatr Otorhinolaryngol, 79, 1393, 10.1016/j.ijporl.2015.05.032 · DOI
- 2014, Effect of separate sampling on classification accuracy, Bioinformatics, 30, 242, 10.1093/bioinformatics/btt662 · DOI
- 2010, Understanding the difficulty of training deep feedforward neural networks, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249
- 1988, Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach, Biometrics, 44, 837, 10.2307/2531595 · DOI · Paper page
- 2016, Learning deep features for discriminative localization, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2921
- 2003, Radiologic correlates of symptom-based diagnostic criteria for chronic rhinosinusitis, Otolaryngol Head Neck Surg, 128, 489
- 2008, Effective doses in radiology and diagnostic nuclear medicine: a catalog, Radiology, 248, 254, 10.1148/radiol.2481071451 · DOI · Paper page
- 2009, Revisiting the interpretation of positive sinus CT findings: a radiological and symptom-based review, Otolaryngol Head Neck Surg, 140, 306, 10.1016/j.otohns.2008.12.007 · DOI
- 2005, Lack of significant correlation between rhinosinusitis symptoms and specific regions of sinus computer tomography scans, Am J Rhinol, 19, 382, 10.1177/194589240501900411 · DOI
Cited by
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
Khanagar, S. B., Al-ehaideb, A., Maganur, P. C., Vishwanathaiah, S., Patil, S., Baeshen, H. A., Sarode, S. C., & Bhandi, S. (2021) · Journal of Dental Sciences · doi:10.1016/j.jds.2020.06.019 · Cited by 676
External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review
Yu, A. C., Mohajer, B., & Eng, J. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210064 · Cited by 408
Malik, H., Farooq, M. S., Khelifi, A., Abid, A., Nasir Qureshi, J., & Hussain, M. (2020) · IEEE Access · doi:10.1109/access.2020.3004766 · Cited by 104
Metadata
- Authors
- Kim, Y., Lee, K. J., Sunwoo, L., Choi, D., Nam, C.-M., Cho, J., Kim, J., Bae, Y. J., Yoo, R.-E., Choi, B. S., Jung, C., & Kim, J. H.
- Venue
- Investigative Radiology
- Published
- 2019-01
- Volume
- 54
- Issue
- 1
- Pages
- 7-15
- ISSN
- 0020-9996 (ISSN-L)
- 1536-0210
- OpenAlex
- W2887005583
- Cited by
- 109
- References
- 10
- FWCI
- 8.06
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep Learning in Diagnosis of Maxillary Sinusitis Using Conventional Radiography. https://app.cochat.ai/papers/10.1097/rli.0000000000000503
- The paper
- Kim, Y., Lee, K. J., Sunwoo, L., Choi, D., Nam, C.-M., Cho, J., Kim, J., Bae, Y. J., Yoo, R.-E., Choi, B. S., Jung, C., & Kim, J. H. (2019). Deep Learning in Diagnosis of Maxillary Sinusitis Using Conventional Radiography. Investigative Radiology. https://doi.org/10.1097/rli.0000000000000503
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.