CoChat
Sign Up
Cited by 5,759AI in cancer detectionAdvanced Neural Network ApplicationsRadiomics and Machine Learning in Medical Imaging

Deep convolutional neural networks for computer-aided detection: cnn architectures, dataset characteristics and transfer learning

Open access
Shin, Hoo-Chang, Roth, Holger R., Gao, Mingchen, Lu, Le, Xu, Ziyue, Nogues, Isabella, Yao, Jianhua, Mollura, Daniel +1 more·2016·IEEE Transactions on Medical Imaging
DOI 10.1109/tmi.2016.2528162PMC PMC4890616
Open this paper in CoChat — free account

Abstract

Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and deep convolutional neural networks (CNNs). CNNs enable learning data-driven, highly representative, hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper empirically evaluates CNN architectures, dataset characteristics, and ImageNet transfer learning for two computer-aided detection tasks: thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. It reports state-of-the-art mediastinal LN detection and the first five-fold cross-validation ILD slice classification results, finding that transfer learning and end-to-end training generally outperform off-the-shelf CNN features.

Key findings

  • For mediastinal LN detection, GoogLeNet-RI-L achieved AUC = 0.95 and 0.85 TPR/3FP, surpassing prior results of AUC = 0.92 and 0.70 TPR/3FP [22] and 0.78 TPR/3FP [36].
  • Transfer learning from ImageNet consistently improved performance: GoogLeNet-TL-H versus GoogLeNet-RI-H gave 0.81 versus 0.61 TPR/3FP in mediastinum and 0.70 versus 0.48 TPR/3FP in abdomen; AlexNet-TL-H versus AlexNet-RI-H gave 0.81 versus 0.79 TPR/3FP in mediastinum and 0.69 versus 0.67 TPR/3FP in abdomen.
  • For ILD slice-level five-fold CV, GoogLeNet-TL reached 0.57 accuracy versus GoogLeNet-RI 0.41, and AlexNet-TL reached 0.46 versus AlexNet-RI 0.44; patch-level ILD classification was less challenging, with five deep models in the range [0.74, 0.76] and an averaged model at 0.79.
  • CNNs trained from scratch or fine-tuned from ImageNet consistently outperformed CNNs using only off-the-shelf CNN features in both LN and ILD problems.

Methods

  • The study evaluated CifarNet, AlexNet, and GoogLeNet architectures (later also OverFeat and VGGNet) on two CADe tasks: thoraco-abdominal LN detection using 388 mediastinal LNs in 90 patient CT scans and 595 abdominal LNs in 86 patient CT scans, and ILD classification using 905 annotated image slices from 120 patients with six ILD labels.
  • LN detection used randomly sampled 2.5D views with fields-of-view spanning 35–128 mm, rescaled to 64×64 pixels, and three-fold cross-validation with FROC AUC and TPR/3FP metrics; ILD classification used slice-level and patch-level tasks under patient-level five-fold cross-validation, with CT windowing encoded into RGB channels and data augmentation.
  • Training used the Caffe framework and NVidia K40 GPUs; models trained from scratch used random Gaussian initialization for 30 epochs with mini-batch size 50, while transfer learning fine-tuned all layers except the last at a learning rate 10 times smaller than default, with the last fully connected layer randomly initialized and trained at 0.01.

Limitations

  • The paper states that the ILD dataset is partially labeled, with very few CT slices labeled as normal or healthy, and that this is one of its main limitations; it also states that the dataset’s limited patient population of 120 patients may not adequately represent ILD disease CT lung imaging patterns.
  • The paper states that slice-level CNN learning is restricted to only 905 CT image slices, that no existing work has reached performance requirements for a realistic clinical setting, and that the most critical bottlenecks for ILD classification are cross-dataset learning and limited patient population size.
  • Evident from the reported results: ILD slice-level CV5 accuracies were low (0.41–0.57), and LOO validation produced much higher accuracies than CV5 (e.g., AlexNet-TL 0.867 versus 0.46; GoogLeNet-TL 0.902 versus 0.57), indicating sensitivity to evaluation protocol; also, for abdominal LN detection the best reported CNN result (0.70 TPR/3FP) did not surpass the prior [22] result of 0.83 TPR/3FP.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tmi.2016.2528162

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 71 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The paper reports the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories.

    “We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories.”
    • Also stated in: § Introduction · plain
      “For the first time, ILD classification results under the patient-level five-fold cross-validation protocol (CV5) are investigated and reported.”
    Quote in the paperSubstantial claim
  • Findingown data

    Different CNN architectures and dataset re-sampling protocols are critical for LN detection tasks where the amount of labeled training data is sufficient and spatial contexts are local.

    “Our experimental results suggest that different CNN architectures and dataset re-sampling protocols are critical for the LN detection tasks where the amount of labeled training data is sufficient and spatial contexts are local.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    LN datasets contain more training and testing data instances than ILD datasets due to data augmentation.

    “As a result, LN datasets contain more training and testing data instances (due to data auugmentation) than ILD datasets.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the publicly available ILD dataset, very few CT slices are labeled as normal or healthy.

    “In the publicly available ILD dataset, very few CT slices are labeled as normal or healthy.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    CNN training learns to ignore areas outside the lung by setting very small filter weights around the corresponding regions.

    “CNN training learns to ignore them by setting very small filter weights around the corresponding regions (Fig. 13).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Some areas outside the lung appear in both healthy or diseased images.
  • May combine two claims. 32
  • Classified as probably not a substantial claim. 28
  • The quote only partly states this claim. 3
  • Borderline: one of the tests for a substantial claim is unclear. 1

  • May combine two claims.

    Findingown data

    The paper's method achieves 86% sensitivity at 3 false positives per patient on mediastinal LN detection, surpassing prior state-of-the-art sensitivities of 78% and 70%.

    “We obtain 86% sensitivity on 3 false positives (FP) per patient, versus the prior state-of-art sensitivities of 78% [36] (stacked shallow learning) and 70% [22] (CNN), as prior state-of-the-art.”
    • Also stated in: § Abstract · plain
      “We achieve the state-of-the-art performance on the mediastinal LN detection”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [36] Seff A., Lu L., Barbu A., Roth H., Shin H.-C., and Summers R. M., “ Leveraging mid-level semantic boundary cues for computer-aided lymph node detection,” Proc. MICCAI, 2015. source

      Not checked in the free run.

    • cites [22] Roth H., “ Improving computer-aided detection using convolutional neural networks and random view aggregation,” IEEE Trans. Med. Imag.. PMC7340334 26441412 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    LN datasets remain less comprehensive than natural image datasets such as ImageNet.

    “They nonetheless remain less comprehensive than natural image datasets, such as ImageNet.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Fine-tuning ImageNet-trained models for ILD classification is advantageous and yields early promising results when the amount of labeled training data is highly insufficient and multi-class categorization is used, as opposed to the LN dataset's binary class categorization.

    “Fine-tuning ImageNet-trained models for ILD classification is clearly advantageous and yields early promising results, when the amount of labeled training data is highly insufficient and multi-class categorization is used, as opposed to the LN dataset's binary class categorization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Random image resampling and reformatting of LN images is a natural choice and yields high CNN performance.

    “Hence, the purely random image resampling (with respect to scale, displacement and orientation) and reformatting (the axial, coronal, and sagittal views are in any system randomly resampled coordinates) is a natural choice, which also happens to yield high CNN performance.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Our best performing CNN model is GoogLeNet-TL (256×256), which obtains AUC=0.92 and 0.70 TPR/3FP.

    “GoogLeNet-TL-H | 0.94 | 0.81 | 0.92 | 0.70”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    GoogLeNet-TL-H yields results similar to AlexNet-TL-H's for mediastinal LN detection, and slightly outperforms Alex-Net-H for abdominal LN detection.

    “GoogLeNet-TL-H yields results similar to AlexNet-TL-H's for the mediastinal LN detection, and slightly outperforms Alex-Net-H for abdominal LN detection.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    AlexNet-TL-H consistently outperforms AlexNet-RI-H, though by smaller margins, with 0.81 versus 0.79 TPR/3FP in mediastinum and 0.69 versus 0.67 TPR/3FP in abdomen.

    “Likewise, AlexNet-TL-H consistently outperforms AlexNet-RI-H, though by smaller margins (0.81 versus 0.79 TPR/3FP in mediastinum; 0.69 versus 0.67 TPR/3FP in abdomen).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Transfer learning from ImageNet pre-trained model is consistently beneficial, as evidenced by AlexNet-TL (0.46) versus AlexNet-RI (0.44), and GoogLeNet-TL (0.57) versus GoogLeNet-RI (0.41).

    “However, transfer learning from ImageNet pre-trained model is consistently beneficial, as evidenced by AlexNet-TL (0.46) versus AlexNet-RI (0.44), and GoogLeNet-TL (0.57) versus GoogLeNet-RI (0.41).”
    Quote in the paperSubstantial claim

  • Borderline on S3 (load-bearing), S4 (based): unclear from the text.

    Findingnone given

    AlexNet-RI-H exhibits less severe over-fitting than GoogLeNet-RI-H.

    “AlexNet-RI-H exhibits less severe over-fitting than GoogLeNet-RI-H.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Many of our CNN models achieve notably better FROC-AUC and TPR/3FP results than previous state-of-the-art models for mediastinal LN detection; GoogLeNet-RI-L obtains AUC=0.95 and 0.85 TPR/3FP.

    “GoogLeNet-RI-L | 0.95 | 0.85 | 0.91 | 0.69”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [36] Seff A., Lu L., Barbu A., Roth H., Shin H.-C., and Summers R. M., “ Leveraging mid-level semantic boundary cues for computer-aided lymph node detection,” Proc. MICCAI, 2015. source

      Not checked in the free run.

    • cites [22] Roth H., “ Improving computer-aided detection using convolutional neural networks and random view aggregation,” IEEE Trans. Med. Imag.. PMC7340334 26441412 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    All five deep models evaluated obtain comparable results within the range of classification accuracy rates [0.74,0.76].

    “Patch-CV5 | 0.76 | 0.74 | 0.76 | 0.75 | 0.76 | 0.79”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Patch-CV5 yields higher F-scores than [54] and classifies the extra consolidation (CD) class.

    “Patch-CV5 yields higher F-scores than [54] and classifies the extra consolidation (CD) class.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [54] Li Q., “ Medical image classification with convolutional neural network,” Proc. 13th Int. Conf. Control Automat. Robot. Vis., pp. 844– 848, 2014.

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    AlexNet-TL and GoogLeNet-TL have consistently better classification accuracies than AlexNet-RI and GoogLeNet-RI, respectively.

    “The quantitative results in Table III indicate that AlexNet-TL and GoogLeNet-TL have consistently better classification accuracies than AlexNet-RI and GoogLeNet-RI, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In ILD slice-level classification, accuracy increases from 0.46 to 0.867 using AlexNet-TL, and from 0.57 to 0.902 for GoogLeNet-TL when comparing CV-5 to LOO.

    “For example, in ILD slice-level classification, the accuracy level drastically increases from 0.46 to 0.867 using AlexNet-TL, and from 0.57 to 0.902 for GoogLeNet-TL.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For AlexNet-RI, training loss significantly decreases as the number of training epochs increases, while validation loss notably increases and validation accuracy does not improve much before reaching a plateau.

    “For AlexNet-RI in Fig. 11(a), the training loss significantly decreases as the number of training epochs increases, while the validation loss notably increases and the validation accuracy does not improve much before reaching a plateau.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    During training, validation loss with transfer learning remains at lower values throughout and achieves higher final accuracy levels than validation loss on a similar problem with random initialization.

    “During the process, the validation loss, having remained at lower values throughout, achieves higher final accuracy levels than the validation loss on a similar problem with random initialization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    LN images are more flexible than ILD images with respect to resampling and reformatting, so LN datasets may be more readily augmented by such image transformations.

    “Since LN images are more flexible than ILD images with respect to resampling and reformatting, LN datasets may be more readily augmented by such image transformations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Three major techniques successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf CNN features, and unsupervised pre-training with supervised fine-tuning.

    “There are currently three major techniques that successfully employ CNNs to medical image classification: 1) training the “CNN from scratch” [13]–[17]; 2) using “off-the-shelf CNN” features (without retraining the CNN) as complementary information channels to existing hand-crafted image features, for chest X-rays [10] and CT lung nodule identification [9], [12]; and 3) performing unsupervised pre-”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [13]–[17]

      Not checked in the free run.

    • cites [9], [12]

      Not checked in the free run.

    • cites [18]–[21]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ImageNet is currently the largest image dataset among other standard datasets for visual recognition.

    “ImageNet is currently the largest image dataset among other standard datasets for visual recognition.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Deng J., “ ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248– 255.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The significant intra-class variation in ImageNet poses greater challenges to any data-driven learning system that builds a classifier to fit given data and generalize to unseen data.

    “This significant intra-class variation poses greater challenges to any data-driven learning system that builds a classifier to fit given data and generalize to unseen data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Transforming axial, coronal, and sagittal representations to RGB facilitates transfer learning from CNN models trained on ImageNet.

    “Transforming axial, coronal, and sagittal representations to RGB also facilitates transfer learning from CNN models trained on ImageNet.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GoogLeNet introduces an Inception module that concatenates filters of different sizes and dimensions into a single new filter.

    “More importantly, it also introduces a new module called “Inception”, which concatenates filters of different sizes and dimensions into a single new filter (refer to Fig. 6).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Szegedy C., “ Going deeper with convolutions,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1– 9. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Evaluation protocols and details are critical to deriving significant empirical findings.

    “Evaluation protocols and details are critical to deriving significant empirical findings [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Chatfield K., Simonyan K., Vedaldi A., and Zisserman A., “ Return of the devil in the details: Delving deep into convolutional nets,” Br. Mach. Vis. Conf., 2014. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AlexNet achieved significantly improved performance over other non-deep learning methods for ILSVRC 2012.

    “The AlexNet architecture was published in [4], achieved significantly improved performance over the other non-deep learning methods for ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Krizhevsky A., Sutskever I., and Hinton G. E., “ ImageNet classification with deep convolutional neural networks,” Proc. NIPS, pp. 1097– 1105, 2012. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The assumption that there are or must be pixel-wise spatial correlations among input channels does not apply to the CNN model representation.

    “In summary, the assumption that there are or must be pixel-wise spatial correlations among input channels does not apply to the CNN model representation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The ImageNet dataset is publicly available, and the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has become the standard benchmark for large-scale object recognition.

    “The ImageNet dataset is publicly available, and the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has become the standard benchmark for large-scale object recognition.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Deng J., “ ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248– 255.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Characterizing an arbitrary CT slice against any possible ILD type without manual ROI can be useful for large-scale patient screening.

    “Characterizing an arbitrary CT slice against any possible ILD type, without any manual ROI (in contrast to [38], [39]), can be useful for large-scale patient screening.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    AlexNet can be fine-tuned on the PASCAL dataset to surpass the performance of the ImageNet pre-trained AlexNet, although the difference is not as significant as that between CNN and non-CNN methods.

    “AlexNet can be fine-tuned on the PASCAL dataset to surpass the performance of the ImageNet pre-trained AlexNet, although the difference is not as significant as that between the CNN and non-CNN methods.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Girshick R., Donahue J., Darrell T., and Malik J., “ Region-based convolutional networks for accurate object detection and semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 1, pp. 142– 158, Jan. 2016. 26656583 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Slice-level ILD classification is a very challenging task where CNN models need to learn from very small numbers of training examples and predict ILD labels on unseen patients.

    “Slice-level ILD classification is a very challenging task where CNN models need to learn from very small numbers of training examples and predict ILD labels on unseen patients.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GoogLeNet is the current state-of-the-art CNN architecture for the ILSVRC challenge, achieving 5.5% top-5 classification error on ImageNet, compared to AlexNet's 15.3% top-5 classification error.

    “GoogLeNet is the current state-of-the-art CNN architecture for the ILSVRC challenge, where it achieved 5.5% top-5 classification error on the ImageNet challenge, compared to AlexNet's 15.3% top-5 classification error.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Szegedy C., “ Going deeper with convolutions,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1– 9. source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Mainstream deep CNN architectures such as AlexNet and GoogLeNet contain tens of millions of free parameters to train, and thus require sufficiently large numbers of labeled medical images.

    “On the other hand, the mainstream deep CNN architectures (e.g., AlexNet and GoogLeNet) contain tens of millions of free parameters to train, and thus require sufficiently large numbers of labeled medical images.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Collecting and annotating large numbers of medical images still poses significant challenges.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ImageNet has more than 1.2 million 256×256 images categorized under 1000 object class categories.

    “ImageNet [1] has more than 1.2 million 256×256 images categorized under 1000 object class categories.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Deng J., “ ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248– 255.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    CT windowing processes do not depend on lung segmentation, which is directly defined in CT HU space.

    “More importantly, these CT windowing processes do not depend on the lung segmentation, which instead is directly defined in the CT HU space.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    ILD classification was evaluated with five-fold cross-validation on patient-level split, as it is more informative for real clinical performance than LOO.

    “We evaluate the ILD classification task with five-fold CV on patient-level split, as it is more informative for real clinical performance than LOO.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The experimental results in [40] are conducted with a patient-level hard split of 100 training and 20 testing.

    “The experimental results in [40] are conducted with a patient-level hard split of 100 (training) and 20 (testing).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Gao M., “ Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks,” in Proc. MICCAI First Workshop Deep Learn. Med. Image Anal., 2015. PMC5881940 29623248

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The difference in performance lies in the fact that annotated LN segmentation masks are required to learn a mid-level semantic boundary detector, whereas CNN approaches only need LN locations for training.

    “This difference lies in the fact that annotated lymph node segmentation masks are required to learn a mid-level semantic boundary detector [36], whereas CNN approaches only need LN locations for training [22].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [36] Seff A., Lu L., Barbu A., Roth H., Shin H.-C., and Summers R. M., “ Leveraging mid-level semantic boundary cues for computer-aided lymph node detection,” Proc. MICCAI, 2015. source

      Not checked in the free run.

    • cites [22] Roth H., “ Improving computer-aided detection using convolutional neural networks and random view aggregation,” IEEE Trans. Med. Imag.. PMC7340334 26441412 source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Both [38] and [39] use leave-one-patient-out (LOO), which is arguably much easier and not directly comparable to 10-fold CV [54] or our Patch-CV5.

    “Both [38] and [39] use the evaluation protocol of “leave-one-patient-out” (LOO), which is arguably much easier and not directly comparable to 10-fold CV [54] or our Patch-CV5.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [38] Song Y., Cai W., Zhou Y., and Feng D. D., “ Feature-based image patch approximation for lung tissue classification,” IEEE Trans. Med. Imag., vol. 32, no. 4, pp. 797– 808, Apr. 2013. 23340591 source

      Not checked in the free run.

    • cites [39] Song Y., “ Large margin local estimate with applications to medical image classification,” IEEE Trans. Med. Imag., vol. 34, no. 6, pp. 1362– 1377, Jun. 2015. 25616009 source

      Not checked in the free run.

    • cites [54] Li Q., “ Medical image classification with convolutional neural network,” Proc. 13th Int. Conf. Control Automat. Robot. Vis., pp. 844– 848, 2014. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    CNNs were trained and evaluated using three-fold cross-validation with folds split into disjoint sets of patients.

    “We train and evaluate CNNs using three-fold cross-validation (folds are split into disjoint sets of patients), with the different CNN architectures described above.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    AlexNet is the first notably successful CNN architecture on the ImageNet challenge and has rekindled significant research interests on CNN.

    “AlexNet is the first notably successful CNN architecture on the ImageNet challenge and has rekindled significant research interests on CNN.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    GoogLeNet is the state-of-the-art deep model, which has outperformed other notable models, such as AlexNet, OverFeat, and VGGNet, in various computer vision benchmarks.

    “GoogLeNet is the state-of-the-art deep model, which has outperformed other notable models, such as AlexNet, OverFeat, and VGGNet [67], [68] in various computer vision benchmarks.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [67] Sermanet P., “ Overfeat: Integrated recognition, localization and detection using convolutional networks,” Proc. ICLR, 2014. source

      Not checked in the free run.

    • cites [68] Simonyan K. and Zisserman A., “ Very deep convolutional networks for large-scale image recognition,” Proc. ICLR, 2014. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    This image sampling scheme contributes to our best reported performance results in both mediastinal LN detection in this paper and automated pancreas segmentation.

    “This image sampling scheme contributes to our best reported performance results in both mediastinal LN detection (in this paper) and automated pancreas segmentation [45].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [45] Roth H., “ DeepOrgan: Multi-level deep convolutional networks for automated pancreas segmentation,” Proc. MICCAI, 2015. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The overall reported LN detection results are clinically significant, as indicated in [63].

    “The overall reported LN detection results are clinically significant, as indicated in [63].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] Kim L., “ Performance assessment of retroperitoneal lymph node computer-assisted detection using random forest and deep convolutional neural network learning algorithms in tandem,” in 102nd Annu. Meet. Radiol. Soc. N. Am., 2014.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationcitation

    Patch-level ILD classification is less challenging than slice-level classification, as far more data samples can be sampled from manually annotated ROIs, up to 100 image patches per ROI.

    “In general, patch-level ILD classification is less challenging than slice-level classification, as far more data samples can be sampled from the manually annotated ROIs (up to 100 image patches per ROI), available from [37].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] Depeursinge A., “ Building a reference multimedia database for interstitial lung diseases,” Comput. Med. Imaging and. Grap., vol. 36, no. 3, pp. 227– 238, 2012. 21803548

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    Gao et al. [40] have argued that a new CADe protocol on holistic classification of ILD diseases directly, using axial CT slice attenuation patterns and CNN, may be more realistic for clinical applications.

    “Recently, Gao et al. [40] have argued that a new CADe protocol on holistic classification of ILD diseases directly, using axial CT slice attenuation patterns and CNN, may be more realistic for clinical applications.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Gao M., “ Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks,” in Proc. MICCAI First Workshop Deep Learn. Med. Image Anal., 2015. PMC5881940 29623248

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OverFeat is described in [67] as an integrated framework for using CNN for classification, localization and detection.

    “OverFeat is described in [67] as an integrated framework for using CNN for classification, localization and detection.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] Sermanet P., “ Overfeat: Integrated recognition, localization and detection using convolutional networks,” Proc. ICLR, 2014.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisenone given

    The main difference between our dataset preparation protocol and that from [22] is a more aggressive extraction of random views within a much larger range of FOVs.

    “The main difference between our dataset preparation protocol and that from [22] is a more aggressive extraction of random views within a much larger range of FOVs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [22] Roth H., “ Improving computer-aided detection using convolutional neural networks and random view aggregation,” IEEE Trans. Med. Imag.. PMC7340334 26441412

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    In abdominal LN detection, [22] obtains the best trade-off between its CNN model complexity and sampled data configuration.

    “In abdominal LN detection, [22] obtains the best trade-off between its CNN model complexity and sampled data configuration.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Roth H., “ Improving computer-aided detection using convolutional neural networks and random view aggregation,” IEEE Trans. Med. Imag.. PMC7340334 26441412

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Although the ILD and LTRC datasets were generated and annotated separately, they contain many common disease labels; EM, GG, FB, and MN belong to both datasets and can be jointly trained/tested to form a larger unified dataset.

    “Although the ILD [37] and LTRC datasets [64] (used in [19]) were generated and annotated separately, they contain many common disease labels. For instance, the ILD disease classes emphysema (EM), ground glass (GG), fibrosis (FB), and micronodules (MN) belong to both datasets, and thus can be jointly trained/tested to form a larger and unified dataset.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [37] Depeursinge A., “ Building a reference multimedia database for interstitial lung diseases,” Comput. Med. Imaging and. Grap., vol. 36, no. 3, pp. 227– 238, 2012. 21803548 source

      Not checked in the free run.

    • cites [64] Holmes III D., Bartholmai B., Karwoski R., Zavaletta V., and Robb R., “ The lung tissue research consortium: An extensive open database containing histological, clinical, and radiological data to study chronic lung disease,” in MICCAI Open Sci. Workshop, 2006.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Abdominal LNs are surrounded by many other similar looking objects, while mediastinal LNs are more easily distinguishable due to larger spatial contexts.

    “As shown in Fig. 1, abdominal LNs are surrounded by many other similar looking objects. Meanwhile, mediastinal LNs are more easily distinguishable, due to the images' larger spatial contexts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    GoogLeNet-RI-H performs poorly because it is susceptible to over-fitting.

    “GoogLeNet-RI-H performs poorly, as it is susceptible to over-fitting.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A segmentation label propagation method was recently proposed to provide full pixel-wise labeling of the ILD data images.

    “A segmentation label propagation method was recently proposed [47] to provide full pixel-wise labeling of the ILD data images.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] Gao M., Lu L., Nogues I., Summers M. R., and Mollura D., “ Segmentation label propagation using deep convolutional neural networks and dense conditional random field,” Proc. IEEE Int. Conf. Comput. Vis., pp. 1377– 1385, 2015. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    For ILD classification, the most critical performance bottlenecks are the challenge of cross-dataset learning and the limited patient population size.

    “For ILD classification, the most critical performance bottlenecks are the challenge of cross-dataset learning and the limited patient population size.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Overfeat is the winning model of the ILSVRC 2013 in detection and classification tasks.

    “Overfeat is the winning model of the ILSVRC 2013 in detection and classification tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] Sermanet P., “ Overfeat: Integrated recognition, localization and detection using convolutional networks,” Proc. ICLR, 2014.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The empirical evaluation, CNN model visualization, CNN performance analysis, and conclusive insights can be generalized to the design of high performance CAD systems for other medical imaging tasks.

    “The empirical evaluation, CNN model visualization, CNN performance analysis, and conclusive insights can be generalized to the design of high performance CAD systems for other medical imaging tasks.”
    Quote in the paperNot a substantial claim
54 of 54

  • The paper relies on this without support.

    Fine-tuning an ImageNet pre-trained CNN model on medical image datasets had not yet been exploited at the time of the study.

    “However, the fine-tuning of an ImageNet pre-trained CNN model on medical image datasets has not yet been exploited.”

  • The paper relies on this without support.

    Obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge.

    “However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge.”

  • The paper relies on this without support.

    Some areas outside the lung appear in both healthy or diseased images.

    “Some areas outside of the lung appear in both healthy or diseased images.”

  • The paper relies on this without support.

    The low-attenuation CT window is useful for visualizing certain texture patterns of lung diseases, especially emphysema.

    “The low-attenuation CT window is useful for visualizing certain texture patterns of lung diseases (especially emphysema).”

  • The paper relies on this without support.

    Collecting and annotating large numbers of medical images still poses significant challenges.

    “Collecting and annotating large numbers of medical images still poses significant challenges.”

  • The authors state this as an assumption.

    OverFeat and VGGNet may generate quantitative performance results ranked between AlexNet's and GoogLeNet's.

    “Likewise, a reasonable assumption is that OverFeat and VGGNet may generate quantitative performance results ranked between AlexNet's and GoogLeNet's.”

  • The authors state this as an assumption.

    OverFeat's and VGGNet's performance levels fall between AlexNet's and GoogLeNet's.

    “The reported results validate our assumption that OverFeat's and VGGNet's performance levels fall between AlexNet's and GoogLeNet's (this observation is consistent with the computer vision findings).”

  • The paper relies on this without support.

    Both the ILD and ImageNet datasets use full-size images.

    “Indeed, both datasets use full-size images.”

  • The paper relies on this without support.

    CifarNet is designed for images with smaller dimensions (32×32 images), and thus is not catered to classification tasks involving 256×256 images.

    “CifarNet is designed for images with smaller dimensions (32×32 images), and thus is not catered to classification tasks involving 256×256 images.”

  • The paper relies on this without support.

    Medical datasets are often biased, in that the number of healthy samples is much larger than the number of diseased instances, or that the numbers of images per class are uneven.

    “Medical datasets are often “biased”, in that the number of healthy samples is much larger than the number of diseased instances, or that the numbers of images per class are uneven.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Transfer learning from natural images has been widely adopted for medical image classification; however the application to medical image segmentation has been scarce.

    “Transfer learning from natural images has been widely adopted for medical image classification Tajbakhsh et al. (2016); Shin et al. (2016); however the application to medical image segmentation has been scarce.”

    From Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation (2020) · cites this paper as Shin et al. (2016)

  • In general, shallow networks have been preferred in medical image analysis, when compared with very deep CNNs employed in computer vision applications.

    “In general, shallow networks have been preferred in medical image analysis, when compared with very deep CNNs employed in computer vision applications [80] [81].”

    From Medical image analysis using convolutional neural networks: a review (2018) · cites this paper as [80]

  • Healthcare applications of deep learning range from one-dimensional biosignal analysis and prediction of medical events to computer-aided detection and diagnosis, drug discovery, therapy selection, operational efficiency, stratified care delivery, and analysis of electronic health records.

    “Healthcare applications of deep learning range from one-dimensional biosignal analysis [17] and the prediction of medical events, e.g. seizures [18] and cardiac arrests [19], to computer-aided detection [20] and diagnosis [21] supporting clinical decision making and survival analysis [22], to drug discovery [23] and as an aid in therapy selection and pharmacogenomics [24], to increased operational”

    From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [20]

  • Dropout regularization was used after the first and second fully connected layers to avoid further overfitting, with a neuron dropout probability of 1-p where p is the probability of neurons kept.

    “After first and second fully connected layers' dropout regularization layer [45] has been used to avoid further overfitting with a neuron dropout probability of 1-p, where p is the probability of neurons kept.”

    From Medical image retrieval using deep convolutional neural network (2017) · cites this paper as [45]

  • A specially designed small network may be more suitable to the smaller-scale training datasets commonly encountered in medical imaging.

    “Instead, a specially designed small network may be more suitable to the smaller-scale training datasets that are commonly encountered in medical imaging [155].”

    From Deep Learning in Medical Ultrasound Analysis: A Review (2019) · cites this paper as [155]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Five-model ensemble lists only four distinct CNNs

    Minor

    The purported list of five CNNs includes AlexNet-RI-H twice, leaving only four distinct model names.

  • GoogLeNet pooling-layer counts disagree

    Minor

    The figure caption says GoogLeNet contains three pooling layers, while the architecture description says it has two pooling layers outside its Inception layers.

  • Table VIII labels its second task as both LN and ILD

    Minor

    The table caption calls the second task LN patch-level detection, but the corresponding column header calls it ILD-Patch.

  • Reported mediastinal sensitivity differs from Table II

    Minor

    The paper reports 86% sensitivity at three false positives per patient, but Table II's best mediastinal TPR/3FP is 0.85, or 85%.

4 of 4

Cites

Flagged first, then by citations

27 of the 104 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. seff, 2015, Leveraging mid-level semantic boundary cues for computer-aided lymph node detection, Proc MICCAI
  2. kim, 2014, Performance assessment of retroperitoneal lymph node computer-assisted detection using random forest and deep convolutional neural network learning algorithms in tandem, 102nd Annu Meet Radiol Soc N Am
  3. sermanet, 2014, Overfeat: Integrated recognition, localization and detection using convolutional networks, Proc ICLR

Cited by

Sorted by citations

Related

Authors
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M.
Affiliations
National Institutes of Health (US)
Corresponding author
Shin, Hoo-Chang
ORCID
Venue
IEEE Transactions on Medical Imaging
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2016-05
Volume
35
Issue
5
Pages
1285-1298
ISSN
  • 0278-0062 (ISSN-L)
  • 1558-254X
DOI
10.1109/tmi.2016.2528162
PMC
PMC4890616
OpenAlex
W2253429366
Cited by
5,759
References
104
Citations by year
2026: 279 · 2025: 480 · 2024: 572 · 2023: 614 · 2022: 725 · 2021: 872 · 2020: 808 · 2019: 712 · 2018: 426 · 2017: 223 · 2016: 44 · 2015: 1 · 2014: 1
FWCI
385.74
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Medical Imaging (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Institutes of Health
  • NIH Clinical Center (Z01CL040004)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
computer-aided detection, deep convolutional neural networks, transfer learning, CNN architecture comparison, medical image classification, ImageNet pre-training, fine-tuning, lymph node detection, interstitial lung disease classification, computed tomography
MeSH terms
Diagnosis, Computer-Assisted, Humans, Image Interpretation, Computer-Assisted, Lymph Nodes, Reproducibility of Results, Databases, Factual, Neural Networks, Computer, Lung Diseases, Interstitial

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review