CoChat
Sign Up
Cited by 433Image Retrieval and Classification TechniquesAdvanced Image and Video Retrieval TechniquesRadiomics and Machine Learning in Medical Imaging

Medical image retrieval using deep convolutional neural network

Open access
Qayyum, Adnan, Anwar, Syed Muhammad, Awais, Muhammad, Majid, Muhammad·2017·Neurocomputing
DOI 10.1016/j.neucom.2017.05.025
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract With a widespread use of digital imaging data in hospitals, the size of medical image repositories is increasing rapidly. This causes difficulty in managing and querying these large databases leading to the need of content based medical image retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low level visual information captured by imaging devices and high level semantic information perceived by human. The efficacy of such systems is more crucial in terms of feature representations that can characterize the high-level information completely. In this paper, we propose a framework of deep learning for CBMIR system by using deep convolutional neural network (CNN) that is trained for classification of medical images. An intermodal dataset that contains twenty-four classes and five modalities is used to train the network. The learned features and the classification results are used to retrieve medical images. For retrieval, best results are achieved when class based predictions are used. An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task. The proposed method is best suited to retrieve multimodal medical images for different body organs.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a content-based medical image retrieval (CBMIR) framework that trains a deep convolutional neural network (DCNN) for 24-class medical image classification, then uses learned fully connected features for retrieval. It reports 99.77% average classification accuracy and a mean average precision of 0.69 when class predictions are used for retrieval.

Key findings

  • The DCNN achieved an average classification accuracy of 99.77%, with AP 99.76, AR 99.77, and F1 99.76 on a 24-class multimodal dataset.
  • Retrieval performance was higher with class prediction: mean average precision was 0.69 with class predictions versus 0.53 without class predictions.
  • Features from the first fully connected layer (FCL1) gave higher precision than features from FCL2 and FCL3.
  • The proposed system performed better than a single-modality organ classification method [43] in Table 2, though the paper notes direct comparison was not possible because no standard medical dataset is available for benchmarking retrieval.

Methods

  • A DCNN with eight layers—five convolutional and three fully connected—was trained for multiclass classification on grayscale 224×224 images, using ReLU nonlinearities, max pooling, dropout, SGD with learning rate 0.0001, and up to 30 epochs.
  • The dataset contained 24 classes and five modalities (MR, CT, PET, PT, OPT), with 300 images per class, totaling 7200 images; 5040 were used for training and 2160 for testing, with no similar images across sets.
  • After training, feature representations were extracted from FCL1–FCL3 and compared with query-image features using Euclidean distance; predicted class labels were optionally used to restrict the search area.

Limitations

  • Stated: the authors note that direct comparison with prior systems was not possible because no standard medical dataset is available to benchmark the retrieval system, and that [48] achieved higher mAP but only for a single modality.
  • Stated: future work is needed to improve retrieval performance using a larger dataset and to adapt the network for 3D volumetric applications.
  • Evident: the dataset uses only 300 images per class and weak global body-part labels, and the best reported retrieval mAP is 0.69, indicating room for improvement despite high classification accuracy.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.neucom.2017.05.025

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 22 cited sources checked · 0 supported · 0 flagged

  • Findingnone given
    § Abstract · p. 1

    For retrieval, the best results are achieved when class-based predictions are used.

    “For retrieval, best results are achieved when class based predictions are used.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    The proposed method achieves a mean average precision of 0.69 for the retrieval task.

    “An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § 5. Conclusion · p. 15

    The network was successfully trained for 24 classes of medical images with an average classification accuracy of 99.77%.

    “The network was successfully trained for 24 classes of medical images with an average classification accuracy of 99.77%.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑚𝑎𝑔𝑒𝑠 𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑚𝑎𝑔𝑒𝑠 𝑖𝑛 𝑑𝑎𝑡𝑎𝑏𝑎𝑠𝑒 . (11) · p. 12

    When class predictions are not used, images from other classes are also retrieved.

    “When class predictions are not used images from other classes are also retrieved highlighted by red boxes in Fig. 9 and Fig. 10.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑚𝑎𝑔𝑒𝑠 𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑚𝑎𝑔𝑒𝑠 𝑖𝑛 𝑑𝑎𝑡𝑎𝑏𝑎𝑠𝑒 . (11) · p. 12

    Precision is high for feature representations of FCL1 as compared to feature representations of layers FCL2 and FCL3.

    “It shows that precision is high for feature representations of FCL1 as compared to feature representations of layer FCL2 and FCL3.”
    Quote in the paperSubstantial claim
  • May combine two claims. 14
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 16

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    The paper's dataset is intermodal, contains twenty-four classes and five modalities, and is used to train the network.

    “An intermodal dataset that contains twenty four classes and five modalities is used to train the network.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § Abstract · p. 1

    The proposed method achieves an average classification accuracy of 99.77%.

    “An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Classification Performance · p. 10

    Each class gives an average accuracy of 100% except stomach, liver and bladder classes, which give an average accuracy of 98.9%, 97.7% and 98.9% respectively.

    “The confusion matrix is shown in Table 1, where each class gives an average accuracy of 100 % except stomach, liver and bladder classes that gives an average accuracy of 98.9%, 97.7% and 98.9% respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Classification Performance · p. 10

    The proposed method achieved an average precision of 99.76, average recall of 99.77, and average F1 measure of 99.76.

    “giving an AP, AR and average F1 measure of 99.76, 99.77 and 99.76 respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 3.1.2 Training Details · p. 6

    With a maximum of 30 epochs of stochastic gradient descent, training the model was optimized with a training error of 0.0422.

    “With a maximum of 30 epochs of SGD, training the model was optimized with a training error of 0.0422.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation
    § 4.4 Comparison · p. 13

    [48] worked for a single modality whereas the proposed system works for multimodal data.

    “Although, [48] achieves a higher value for mAP, but they have only worked for a single modality whereas, our proposed system work for multimodal data.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] He, 2016, Deep residual learning for image recognition

      Not checked in the free run.

  • May combine two claims.

    Findingreasoning
    § 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑚𝑎𝑔𝑒𝑠 𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑚𝑎𝑔𝑒𝑠 𝑖𝑛 𝑑𝑎𝑡𝑎𝑏𝑎𝑠𝑒 . (11) · p. 13

    The system achieves a mean average precision of 0.53 without class predictions and 0.69 with class prediction.

    “The system achieves a mean average precision of 0.53 and 0.69 without class predictions and with class prediction respectively.”
    • Also stated in: § 5. Conclusion · p. 15 · plain
      “The proposed system achieves a mean average precision of 0.69 for multimodal image data with class prediction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 1. Introduction · p. 1

    The semantic gap can be reduced either by embedding domain-specific knowledge or by using a machine learning technique to develop intelligent systems that can be trained to act like the human vision system.

    “This gap can be reduced either by embedding domain specific knowledge or by using some machine learning technique to develop intelligent systems that can be trained to act like HVS.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 1. Introduction · p. 2

    The paper's annotation approach uses weak supervision at the global body-part level, which requires less time for labelling and decreases the annotation effort required in the training phase.

    “In this way, the supervision is very weak and requires very less time for labelling, hence decreasing the annotation effort required in training phase.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Weak supervision at the global body-part level is sufficient to train a DCNN that learns features useful for retrieval.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1.1 The DCNN Model Architecture · p. 6

    A network with rectified linear units can be trained several times faster than its equivalent with tanh units and allows deeper networks with vanishing gradient problems.

    “This network with ReLUs has the ability to get trained several times faster than its equivalent with tanh units [37] as well as it also allows to go deeper with vanishing gradient problems.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] Hahnloser, 2000, Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit, Nature source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § 3.1.1 The DCNN Model Architecture · p. 6

    Dropout regularization was used after the first and second fully connected layers to avoid further overfitting, with a neuron dropout probability of 1-p where p is the probability of neurons kept.

    “After first and second fully connected layers' dropout regularization layer [45] has been used to avoid further overfitting with a neuron dropout probability of 1-p, where p is the probability of neurons kept.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Shin, 2016, Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning, IEEE Trans. Med. Imaging source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1. Introduction · p. 1

    The most challenging issue remaining in content-based image retrieval systems is reducing the semantic gap.

    “Although, many researchers have broadly studied these areas [8], but the most challenging issue that remains in CBIR systems is reducing the 'semantic gap'.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9] Kumar, 2014, A novel approach to self order feature reweighting in CBIR to reduce semantic gap using relevance feedback

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 1. Introduction · p. 2

    Deep learning methods have been applied to the content-based image retrieval task in recent studies, but there is less attention on exploring deep learning methods for the content-based medical image retrieval task.

    “Deep learning methods have been applied to CBIR task in recent studies [8, 17, 18], but there is less attention on exploring deep learning methods for CBMIR task.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [8] Wan, 2014, Deep learning for content-based image retrieval: a comprehensive study

      Not checked in the free run.

    • cites [17] Babenko, 2015, Aggregating local deep features for image retrieval

      Not checked in the free run.

    • cites [18] Lin, 2015, Deep learning of binary hash codes for fast image retrieval

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.3 Deep Learning · p. 4

    Recent studies have shown that deep learning techniques are successfully applied to the medical domain.

    “Recent studies have shown that deep learning techniques are successfully applied to medical domain.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [40] LeCun, 1998, Gradient-based learning applied to document recognition, Proc. IEEE source

      Not checked in the free run.

    • cites [41] Anthimopoulos, 2016, Lung pattern classification for interstitial lung diseases using a deep convolutional neural network, IEEE Trans. Med. Imaging source

      Not checked in the free run.

    • cites [42] van Tulder, 2016, Combining generative and discriminative representation learning for lung CT analysis with convolutional restricted Boltzmann machines, IEEE Trans. Med. Imaging source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction · p. 2

    Recent studies have reported that deep learning methods have been successfully applied to image and video classification, visual tracking, speech recognition, and natural language processing.

    “Recent studies have reported that deep learning methods have been successfully applied to many applications areas e.g., image and video classification [11-13], visual tracking [14], speech recognition [15] and natural language processing [16].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [11] Krizhevsky, 2012, Imagenet classification with deep convolutional neural networks

      Not checked in the free run.

    • cites [12] Zeiler, 2014, Visualizing and understanding convolutional networks

      Not checked in the free run.

    • cites [13] Karpathy, 2014, Large-scale video classification with convolutional neural networks

      Not checked in the free run.

    • cites [14] Wu, 2016, Regional deep learning model for visual tracking, Neurocomputing source

      Not checked in the free run.

    • cites [15] Hinton, 2012, Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups, IEEE Signal Process. Mag. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisereasoning
    § 1. Introduction · p. 2

    For medical imaging, global body-part annotation is useful because annotations usually require expert advice and high cost.

    “For medical imaging, this type of annotation is quite useful since, annotations usually require expert advice and high cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.1 Phase 1: Classification · p. 5

    Because a DCNN learns features directly from images, it can identify the class of a query image more effectively and therefore the learned features can be used for the image retrieval task.

    “Hence, it can identify the class of a query image more effectively and therefore the learned features can be used for image retrieval task.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 4.2 Classification Performance · p. 10

    A 10-fold cross validation was used on the training data and the testing set was tested 100 times.

    “A 10-fold cross validation was used on the training data and the testing set was tested 100 times”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 3.2 Phase 2: Features Extraction for CBMIR · p. 8

    Using the predicted class label to limit the search area in the database reduces the number of computations and eliminates irrelevant images from retrieval results.

    “the predicted class label (dashed line in Fig. 2) has been used to limit the search area in the database by reducing the number of computations and eliminating irrelevant images from retrieval results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 3.1.2 Training Details · p. 6

    Keeping the learning rate constant for all iterations is appropriate for the dataset used because decaying the learning rate increases training time compared with keeping it constant.

    “The learning rate was kept constant for all iterations of stochastic gradient i.e., 0.0001 as for dataset used, decaying the learning rate, increases the training time as compared to keeping it constant.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given
    § 4.1 Dataset · p. 10

    The training and testing sets did not contain any similar image.

    “The training and testing sets did not contain any similar image.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 3.1 Phase 1: Classification · p. 5

    A deep learning algorithm learns low-level, mid-level and abstract features directly from images as opposed to making domain-specific assumptions, which is the case for handcrafted features.

    “Deep learning algorithm learns low-level, mid-level and abstract features directly from the images as opposed to making domain specific assumptions, which is the case for handcrafted features.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given
    § 4.2 Classification Performance · p. 10

    The system trained in [43] is on a different set of image collection.

    “the system trained in [43] is on a different set of image collection”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [43] Moeskops, 2016, Automatic segmentation of MR brain images with a convolutional neural network, IEEE Trans. Med. Imaging source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.1 Dataset · p. 10

    The data from each class was divided randomly into training and testing sets using 70% and 30% of images for training and testing, respectively.

    “The data from each class was divided randomly into training and testing set using 70% and 30% images for training and testing set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 3.1.1 The DCNN Model Architecture · p. 6

    The overlapping pooling operation makes it difficult for the model to overfit during training.

    “The overlapping pooling operation makes it difficult for the model to overfit during training.”
    Quote in the paperNot a substantial claim
25 of 25

  • The paper relies on this without support.

    The 2D slices used in the dataset are adequate to represent the 3D medical images for retrieval.

    “This paper focuses on retrieval of these 2D slices”

  • The paper relies on this without support.

    Global body-part class labels provide a clinically meaningful basis for retrieving similar medical images.

    “the classes were formulated at global level i.e., images from different body parts were divided into separate classes with respective body part label”

  • The paper relies on this without support.

    The collected dataset is representative of multimodal medical images across different body organs.

    “A dataset is carefully collected that is multimodal and covers a wide range of medical imaging target areas.”

  • The paper relies on this without support.

    Weak supervision at the global body-part level is sufficient to train a DCNN that learns features useful for retrieval.

    “In this way, the supervision is very weak and requires very less time for labelling, hence decreasing the annotation effort required in training phase.”

  • The paper relies on this without support.

    No standard medical dataset is available that can be used to benchmark the retrieval system.

    “As direct comparison was not possible, since to the best of our knowledge, no standard medical dataset is available that can be used to benchmark the retrieval system.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Qayyum, A., Anwar, S. M., Awais, M., & Majid, M.
Affiliations
  • University of Engineering and Technology Taxila (PK)
  • University of Surrey (GB)
Corresponding author
Anwar, Syed Muhammad
ORCID
Venue
Neurocomputing
Publisher
Elsevier BV
Published
2017-11
Volume
266
Pages
8-20
ISSN
0925-2312 (ISSN-L)
DOI
10.1016/j.neucom.2017.05.025
OpenAlex
W2601707599
Semantic Scholar
8b612f02676d1fe5da6cc097ff587e242ca777f2
Cited by
433
References
59
Citations by year
2026: 9 · 2025: 33 · 2024: 38 · 2023: 54 · 2022: 77 · 2021: 73 · 2020: 64 · 2019: 44 · 2018: 34 · 2017: 7
FWCI
16.91
Type
journal article
Language
en
Open access
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Neurocomputing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
content-based medical image retrieval, deep convolutional neural networks, semantic gap, multimodal medical images, medical image classification, learned features

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review