Content-based brain tumor retrieval for MR images using transfer learning
Open accessAbstract
This paper presents an automatic content-based image retrieval (CBIR) system for brain tumors on T1-weighted contrast-enhanced magnetic resonance images (CE-MRI). The key challenge in CBIR systems for MR images is the semantic gap between the low-level visual information captured by the MRI machine and the high-level information perceived by the human evaluator. The traditional feature extraction methods focus only on low-level or high-level features and use some handcrafted features to reduce this gap. It is necessary to design a feature extraction framework to reduce this gap without using handcrafted features by encoding/combining low-level and high-level features. Deep learning is very powerful for feature representation that can depict low-level and high-level information completely and embed the phase of feature extraction in self-learning. Therefore, we propose a deep convolutional neural network VGG19-based novel feature extraction framework and apply closed-form metric learning to measure the similarity between the query image and database images. Furthermore, we adopt transfer learning and propose a block-wise fine-tuning strategy to enhance the retrieval performance. The extensive experiments are performed on a publicly available CE-MRI dataset that consists of three types of brain tumors (i.e., glioma, meningioma, and pituitary tumor) collected from 233 patients with a total of 3064 images across the axial, coronal, and sagittal views. Our method is more generic, as we do not use any handcrafted features; it requires minimal preprocessing, tested as robust on fivefold cross-validation, can achieve a fivefold mean average precision of 96.13%, and outperforms the state-of-the-art CBIR systems on the CE-MRI dataset.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Swati et al. proposed a method that classifies the brain tumor into glioma, meningioma, and pituitary tumor with the minimum intervention of radiologists.
“Swati et al. [64] proposed a method that classifies the brain tumor into glioma, meningioma, and pituitary tumor with the minimum intervention of radiologists.”
From A Comprehensive Analysis of Recent Deep and Federated-Learning-Based Methodologies for Brain Tumor Diagnosis (2022) · cites this paper as [64]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759 · see citing works
Deep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968 · see citing works
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,651 · see citing works
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451 · see citing works
Brain tumor segmentation with Deep Neural Networks
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.05.004 · Cited by 3,278 · see citing works
Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images
Pereira, S., Pinto, A., Alves, V., & Silva, C. A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2538465 · Cited by 2,735 · see citing works
Distance Metric Learning with Application to Clustering with Side-Information
Xing, E. P., Jordan, M. I., Russell, S., & Ng, A. Y. (2002) · Cited by 2,700 · see citing works
The PASCAL visual object classes challenge 2006 (VOC2006) results
Everingham, M., Zisserman, A., Williams, C. K. I., & Gool, L. V. (2006) · Cited by 2,079 · see citing works
Greenspan, H. K., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,704 · see citing works
Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · see citing works
Aggregating Local Image Descriptors into Compact Codes
Jegou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., & Schmid, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.235 · Cited by 1,503 · see citing works
Hierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968 · see citing works
Deep convolutional neural networks for multi-modality isointense infant brain image segmentation
Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., & Shen, D. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.061 · Cited by 817 · see citing works
Deep Learning for Content-Based Image Retrieval
Wan, J., Wang, D., Hoi, S. C. H., Wu, P., Zhu, J., Zhang, Y., & Li, J. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654948 · Cited by 814 · see citing works
Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?
Penatti, O. A. B., Nogueira, K., & dos Santos, J. A. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301382 · Cited by 730 · see citing works
Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks
Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., Mok, V. C., Shi, L., & Heng, P.-A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528129 · Cited by 679 · see citing works
Deep MRI brain extraction: A 3D convolutional neural network for skull stripping
Kleesiek, J., Urban, G., Hubert, A., Schwarz, D., Maier-Hein, K., Bendszus, M., & Biller, A. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.01.024 · Cited by 505 · see citing works
Sparse Representation Based Fisher Discrimination Dictionary Learning for Image Classification
Yang, M., Lei Zhang, A., Feng, X., & Zhang, D. D. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0722-8 · Cited by 490 · see citing works
Object Detection Networks on Convolutional Feature Maps
Ren, S., He, K., Girshick, R., Zhang, X., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2601099 · Cited by 454 · see citing works
From generic to specific deep representations for visual recognition
Azizpour, H., Razavian, A. S., Sullivan, J., Maki, A., & Carlsson, S. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301270 · Cited by 401 · see citing works
Learning Local Feature Descriptors Using Convex Optimisation
Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2301163 · Cited by 345 · see citing works
State-space model with deep learning for functional dynamics estimation in resting-state fMRI
Suk, H., Chong-yaw Wee, A., Lee, S., & Shen, D. G. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.01.005 · Cited by 311 · see citing works
Deep learning with non-medical training used for chest pathology identification
Bar, Y., Diamant, I., Wolf, L., & Greenspan, H. K. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2083124 · Cited by 300 · see citing works
Retrieval of Brain Tumors by Adaptive Spatial Pooling and Fisher Vector Representation
Cheng, J., Yang, W., Huang, M., Huang, W. F., Jiang, J., Zhou, Y., Yang, R., Zhao, J., Feng, Y., Feng, Q., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157112 · Cited by 298 · see citing works
Ciompi, F., de Hoop, B., van Riel, S. J., Chung, K., Scholten, E. T., Oudkerk, M., de Jong, P. A., Prokop, M., & Ginneken, B. V. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.08.001 · Cited by 289 · see citing works
van Ginneken, B., Setio, A. A. A., Jacobs, C., & Ciompi, F. (2015) · 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2015.7163869 · Cited by 278 · cited 2× · see citing works
“This transfer learning and fine-tuning suggested an alternative to [5], [7], [8], [22], [51], and [52], Once the model was successfully optimized and trained, we extracted features of the training and testing datasets from the trained model.”
MethodX-ray Categorization and Retrieval on the Organ and Pathology Level, Using Patch-Based Visual Words
Avni, U., Greenspan, H. K., Konen, E., Sharon, M., & Goldberger, J. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2095026 · Cited by 193 · see citing works
Medical Image Categorization and Retrieval for PACS Using the GMM-KL Framework
Greenspan, H., & Pinhas, A. T. (2007) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2006.874191 · Cited by 184 · see citing works
Deep Contextual Networks for Neuronal Structure Segmentation
Chen, H., Qi, X., Cheng, J., & Heng, P. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10141 · Cited by 152 · see citing works
van Tulder, G., & de Bruijne, M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2526687 · Cited by 142 · see citing works
Capturing image semantics with low-level descriptors
Mojsilović, A., & Rogowitz, B. E. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2001.958942 · Cited by 126 · see citing works
Computer-Aided Diagnosis of Mammographic Masses Using Scalable Image Retrieval
Jiang, M., Zhang, S., Li, H., & Metaxas, D. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2365494 · Cited by 114 · see citing works
Rahman, M. M., Desai, B. C., & Bhattacharya, P. (2007) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2007.10.001 · Cited by 98 · see citing works
Jiang, J., Wu, Y., Huang, M., Yang, W., Chen, W., & Feng, Q. (2013) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2013.05.007 · Cited by 97 · see citing works
A Pattern Similarity Scheme for Medical Image Retrieval
Iakovidis, D. K., Pelekis, N., Kotsifakos, E. E., et al. (2009) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2008.923144 · Cited by 89 · see citing works
Yang, W., Lu, Z., Yu, M., Huang, M., Feng, Q., & Chen, W. (2012) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-012-9495-1 · Cited by 83 · see citing works
Brain Tumor Classification Using Wavelet and Texture Based Neural Network
John, P. (2012) · Cited by 82 · see citing works
Online Multi-Modal Distance Metric Learning with Application to Image Retrieval
Wu, P., Hoi, S. C. H., Zhao, P., et al. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2477296 · Cited by 77 · see citing works
Liu, X., Tizhoosh, H. R., & Kofman, J. (2016) · doi:10.1109/ijcnn.2016.7727562 · Cited by 65 · see citing works
Anavi, Y., Kogan, I., Gelbart, E., et al. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2217587 · Cited by 60 · see citing works
Yang, W., Feng, Q., Yu, M., Lu, Z., Gao, Y., Xu, Y., & Chen, W. (2012) · Medical Physics · doi:10.1118/1.4754305 · Cited by 49 · see citing works
Distance metric learning vs. Fisher discriminant analysis
Alipanahi, B., Biggs, M., & Ghodsi, A. (2008) · National Conference on Artificial Intelligence · Cited by 48 · see citing works
Deep Learning in Diagnosis of Brain Disorders
Suk, H., Shen, D. G., & Initiative, A. D. N. (2015) · Trends in augmentation of human performance · doi:10.1007/978-94-017-7239-6_14 · Cited by 46 · see citing works
Huang, M., Yang, W., Yu, M., et al. (2012) · Computational and Mathematical Methods in Medicine · doi:10.1155/2012/280538 · Cited by 44 · see citing works
Huang, M., Yang, W., Wu, Y., Jiang, J., Gao, Y., Chen, Y., Feng, Q., Chen, W., & Lu, Z. (2014) · PLoS ONE · doi:10.1371/journal.pone.0102754 · Cited by 40 · see citing works
Deeply learnt hashing forests for content based image retrieval in prostate MR images
Shah, A. M., Conjeti, S., Navab, N., & Katouzian, A. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2217162 · Cited by 25 · see citing works
Bridging the semantic gap using Ranking SVM for image retrieval
Guan, H., Antani, S., Long, L. R., & Thoma, G. R. (2009) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2009.5193057 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A scoping review of transfer learning research on medical image analysis using ImageNet
Morid, M. A., Borjali, A., & Del Fiol, G. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104115 · Cited by 508 · see citing works
Transfer learning in magnetic resonance brain imaging: a systematic review
Valverde, J. M., Imani, V., Abdollahzadeh, A., De Feo, R., Prakash, M., Ciszek, R., & Tohka, J. (2021) · Journal of Imaging · doi:10.3390/jimaging7040066 · Cited by 137 · see citing works
Naeem, A., Anees, T., Naqvi, R. A., & Loh, W.-K. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12020275 · Cited by 103 · see citing works
- Authors
- Swati, Z. N. K., Zhao, Q., Kabir, M., Ali, F., Ali, Z., Ahmed, S., & Lu, J.
- Affiliations
- Nanjing University of Science and Technology (CN)
- Nanjing University of Finance and Economics (CN)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 17809-17822
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2910541852
- Cited by
- 246
- References
- 57
- Citations by year
- 2026: 15 · 2025: 41 · 2024: 25 · 2023: 52 · 2022: 41 · 2021: 44 · 2020: 24 · 2019: 4
- FWCI
- 12.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Key Research and Development Program of China (2018YFB1004)
- 111 Project (B13022)
- Natural Science Foundation of Jiangsu Province (20131351)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- content-based image retrieval, brain tumors, contrast-enhanced MRI, transfer learning, VGG19, metric learning, semantic gap, feature extraction, glioma, meningioma, pituitary tumors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
