An ensemble of fine-tuned convolutional neural networks for medical image classification
Open accessAbstract
The availability of medical imaging data from clinical archives, research literature, and clinical manuals, coupled with recent advances in computer vision offer the opportunity for image-based diagnosis, teaching, and biomedical research. However, the content and semantics of an image can vary depending on its modality and as such the identification of image modality is an important preliminary step. The key challenge for automatically classifying the modality of a medical image is due to the visual characteristics of different modalities: some are visually distinct while others may have only subtle differences. This challenge is compounded by variations in the appearance of images based on the diseases depicted and a lack of sufficient training data for some modalities. In this paper, we introduce a new method for classifying medical images that uses an ensemble of different convolutional neural network (CNN) architectures. CNNs are a state-of-the-art image classification technique that learns the optimal image features for a given classification task. We hypothesise that different CNN architectures learn different levels of semantic image representation and thus an ensemble of CNNs will enable higher quality features to be extracted. Our method develops a new feature extractor by fine-tuning CNNs that have been initialized on a large dataset of natural images. The fine-tuning process leverages the generic image features from natural images that are fundamental for all images and optimizes them for the variety of medical imaging modalities. These features are used to train numerous multiclass classifiers whose posterior probabilities are fused to predict the modalities of unseen images. Our experiments on the ImageCLEF 2016 medical image public dataset (30 modalities; 6776 training images, and 4166 test images) show that our ensemble of fine-tuned CNNs achieves a higher accuracy than established CNNs. Our ensemble also achieves a higher accuracy than methods in the literature evaluated on the same benchmark dataset and is only overtaken by those methods that source additional training data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DTL significantly reduces the demand for training data and training time for a target domain-specific task by choosing a pre-trained model for a fixed feature extractor or for further fine-tuning.
“DTL significantly reduces the demand for training data and training time for a target domain-specific task by choosing a pre-trained model (trained on another large dataset of same target domain) for a fixed feature extractor [44] or for further fine-tuning [45].”
From A survey on deep transfer learning to edge computing for mitigating the COVID-19 pandemic (2020) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
Chiang, L. H., Russell, E. L., & Braatz, R. D. (2001) · Advanced textbooks in control and signal processing · doi:10.1007/978-1-4471-0347-9_4 · Cited by 6,684 · see citing works
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759 · see citing works
Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Williams, C. K. I. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s269 · Cited by 4,390 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335 · see citing works
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · see citing works
Pattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,782 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714 · see citing works
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · see citing works
Lung pattern classification for interstitial lung diseases using a deep convolutional neural network
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., & Mougiakakou, S. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535865 · Cited by 1,343 · see citing works
An Ensemble Classification-Based Approach Applied to Retinal Blood Vessel Segmentation
Fraz, M. M., Remagnino, P., Hoppe, A., et al. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2012.2205687 · Cited by 1,039 · see citing works
Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks
Chen, H., Ni, D., Qin, J., Li, S., Yang, X., Wang, T., & Heng, P. A. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2425041 · Cited by 379 · see citing works
Kumar, A., Kim, J., Cai, W., FULHAM, M. J., & Feng, D. D. (2013) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-013-9619-2 · Cited by 227 · see citing works
Self-citationTransfer representation learning for medical image analysis
Shie, C., Chuang, C.-H., Chou, C., Wu, M.-H., & Chang, E. Y. (2015) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2015.7318461 · Cited by 135 · see citing works
Kalpathy–Cramer, J., de Herrera, A. G. S., Demner‐Fushman, D., Antani, S., Bedrick, S. D., & Müller, H. (2014) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2014.03.004 · Cited by 130 · see citing works
Statistical framework for model-based image retrieval in medical applications
Keysers, D., Dahmen, J., Ney, H., et al. (2003) · Journal of Electronic Imaging · doi:10.1117/1.1525790 · Cited by 127 · see citing works
Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition
Margeta, J., Criminisi, A., Lozoya, R. C., Lee, D. C., & Ayache, N. (2015) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2015.1061448 · Cited by 94 · see citing works
Transfer learning of a convolutional neural network for HEp-2 cell image classification
Phan, H. T. H., Kumar, A., Kim, J., & Feng, D. D. (2016) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2016.7493483 · Cited by 74 · see citing works
Self-citationReducing multiclass to binary by coupling probability estimates
Zadrozny, B. (2001) · Cited by 71 · see citing works
Improved medical image modality classification using a combination of visual and textual features
Dimitrovski, I., Kocev, D., Kitanovski, I., et al. (2014) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2014.06.005 · Cited by 65 · see citing works
General Overview of ImageCLEF at the CLEF 2015 Labs
Villegas, M., Müller, H., Gilbert, A., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24027-5_45 · Cited by 53 · see citing works
Comparative object similarity for improved recognition with few or no examples
Wang, G., Forsyth, D. A., & Hoiem, D. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539955 · Cited by 51 · see citing works
Overview of the ImageCLEF 2016 Medical Task.
de Herrera, A. G. S., Schaer, R., Bromuri, S., & Müller, H. (2016) · CLEF (Working Notes) · Cited by 41 · see citing works
Kumar, A., Sridar, P., Quinton, A. E., Kumar, R. K., Feng, D. D., Nanan, R. K. H., & Kim, J. (2016) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2016.7493385 · Cited by 40 · see citing works
Self-citationKoitka, S., & Friedrich, C. M. (2016) · CLEF (Working Notes) · Cited by 33 · see citing works
A generalized framework for medical image classification and recognition
Abedini, M., Codella, N., Connell, J. H., Garnavi, R., Merler, M., Pankanti, S., Smith, J. R., & Syeda-Mahmood, T. F. (2015) · IBM Journal of Research and Development · doi:10.1147/jrd.2015.2390017 · Cited by 31 · see citing works
Multimodal biomedical image retrieval using hierarchical classification and modality fusion
Rahman, M. M., You, D., Simpson, M. S., Antani, S., Demner‐Fushman, D., & Thoma, G. R. (2013) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-013-0038-4 · Cited by 30 · see citing works
FHDO Biomedical Computer Science Group at Medical Classification Task of ImageCLEF 2015.
Pelka, O., & Friedrich, C. M. (2015) · CLEF (Working Notes) · Cited by 25 · see citing works
How do health care professionals select medical images they need?
Sedghi, S., Sanderson, M., & Clough, P. (2012) · Aslib Journal of Information Management · doi:10.1108/00012531211244815 · Cited by 17 · see citing works
Stanley, R. J., De, S., Demner‐Fushman, D., et al. (2010) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2010.11.008 · Cited by 14 · see citing works
FCSE at Medical Tasks of ImageCLEF 2013
Kitanovski, I., Dimitrovski, I., & Loškovska, S. (2013) · Cited by 13 · see citing works
The Key Image and Case Log Application
Rowe, S. P., Siddiqui, A., & Bonekamp, D. (2014) · Academic Radiology · doi:10.1016/j.acra.2014.04.001 · Cited by 12 · see citing works
Convolutional Neural Networks for Medical Clustering
Lyndon, D., Kumar, A., Kim, J., et al. (2015) · CLEF (Working Notes) · Cited by 11 · see citing works
Self-citationSubfigure and Multi-Label Classification using a Fine-Tuned Convolutional Neural Network.
Kumar, A., Lyndon, D., Kim, J., et al. (2016) · CLEF (Working Notes) · Cited by 11 · see citing works
Self-citationGeneral Overview of ImageCLEF at the CLEF 2016 Labs
Villegas, M., Müller, H., de Herrera, A. G. S., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44564-9_25 · Cited by 11 · see citing works
Choi, S. (2015) · CLEF (Working Notes) · Cited by 9 · see citing works
IPL at CLEF 2016 Medical Task.
Valavanis, L., Stathopoulos, S., & Kalamboukis, T. (2016) · CLEF (Working Notes) · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- M. Abedini, L. Cao, N. Codella, J. H. Connell, R. Garnavi, A. Geva, M. Merler, Q.-B. Nguyen, S. U. Pankanti, J. R. Smith, X. Sun, and A. Tzadok, "IBM research at ImageCLEF 2013 medical tasks," in CLEF 2013 Working Notes, ser. CEUR Workshop Proceedings, vol. 1179, 2013.
- D. Lyndon, A. Kumar, J. Kim, P. H. W. Leong, and D. Feng, "Con- volutional neural networks for medical classification," in CLEF 2015 Working Notes, ser. CEUR Workshop Proceedings, vol. 1391, 2015.
- D. Semedo and J. Magalhães, "NovaSearch at ImageCLEFmed 2016 Subfigure Classification Task," in CLEF 2016 Working Notes, ser. CEUR Workshop Proceedings, vol. 1609, 2016, pp. 386-398.
Cited by
Sorted by citations
Dong, X., Yu, Z., Cao, W., Shi, Y., & Ma, Q. (2020) · Frontiers of Computer Science · doi:10.1007/s11704-019-8208-z · Cited by 1,928 · see citing works
A comprehensive review on ensemble deep learning: Opportunities and challenges
Mohammed, A., & Kora, R. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.01.014 · Cited by 1,127 · see citing works
Mazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26534 · Cited by 554 · see citing works
A systematic survey of computer-aided diagnosis in medicine: Past and present developments
Yanase, J., & Triantaphyllou, E. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.112821 · Cited by 303 · see citing works
Hamamoto, R., Suvarna, K., Yamada, M., Kobayashi, K., Shinkai, N., Miyake, M., Takahashi, M., Jinnai, S., Shimoyama, R., Sakai, A., et al. (2020) · Cancers · doi:10.3390/cancers12123532 · Cited by 207 · see citing works
A survey on deep transfer learning to edge computing for mitigating the COVID-19 pandemic
Sufian, A., Ghosh, A., Sadiq, A. S., & Smarandache, F. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2020.101830 · Cited by 156 · see citing works
Alzubaidi, L., Al-Shamma, O., Fadhel, M. A., Farhan, L., Zhang, J., & Duan, Y. (2020) · Electronics · doi:10.3390/electronics9030445 · Cited by 143 · see citing works
Sridar, P., Kumar, A., Quinton, A., Nanan, R., Kim, J., & Krishnakumar, R. (2019) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2018.11.016 · Cited by 70 · see citing works
Self-citationYue, G., Wei, P., Zhou, T., Song, Y., Zhao, C., Wang, T., & Lei, B. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3319516 · Cited by 24 · see citing works
- Authors
- Kumar, A., Kim, J., Lyndon, D., Fulham, M., & Feng, D.
- Affiliations
- The University of Sydney (AU)
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2017-01
- Volume
- 21
- Issue
- 1
- Pages
- 31-40
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W2559785631
- Cited by
- 515
- References
- 44
- Citations by year
- 2026: 18 · 2025: 35 · 2024: 50 · 2023: 52 · 2022: 72 · 2021: 88 · 2020: 83 · 2019: 61 · 2018: 39 · 2017: 17
- FWCI
- 26.49
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion)
- PubMed (publishedVersion)
- The Sydney eScholarship Repository (The University of Sydney) (submittedVersion)
- Funders
- ARC
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- ensemble convolutional neural networks, fine-tuning, feature extraction, medical image classification
- MeSH terms
- Machine Learning, Diagnostic Imaging, Humans, Image Processing, Computer-Assisted, Neural Networks, Computer, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
