Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images
Open accessThe DOI matches its Crossref record; 50 of the 51 references it cites were found in a registry or an index.
Abstract
Convolutional neural networks (CNNs) are deep learning network architectures that have pushed forward the state-of-the-art in a range of computer vision applications and are increasingly popular in medical image analysis. However, training of CNNs is time-consuming and challenging. In medical image analysis tasks, the majority of training examples are easy to classify and therefore contribute little to the CNN learning process. In this paper, we propose a method to improve and speed-up the CNN training for medical image analysis tasks by dynamically selecting misclassified negative samples during training. Training samples are heuristically sampled based on classification by the current status of the CNN. Weights are assigned to the training samples and informative samples are more likely to be included in the next CNN training iteration. We evaluated and compared our proposed method by training a CNN with (SeS) and without (NSeS) the selective sampling method. We focus on the detection of hemorrhages in color fundus images. A decreased training time from 170 epochs to 60 epochs with an increased performance-on par with two human experts-was achieved with areas under the receiver operating characteristics curve of 0.894 and 0.972 on two data sets. The SeS CNN statistically outperformed the NSeS CNN on an independent test set.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Twenty-five studies17,28,29,35,37,39,40,44–61 compared algorithm performance against healthcare professionals.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [44]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“Using multiple smaller sized layers with non-linear rectifications makes the CNN more discriminative and less parameters need to be optimized [30].”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“The CNN architecture used in this study consists of five convolutional layers followed by rectified linear units (ReLUs) [4] and spatial max-pooling.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“Additions such as the inclusion of drop-out [42] or batch normalization [43] could potentially further increase performance.”
BackgroundA decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“Pilot experiments using MSRA weight initialization [3] showed an equal number of epochs required to train the SeS with similar end-performance on the test sets.”
MethodDeep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
(2012) · Cited by 8,009
(2007) · Developments in Ophthalmology · doi:10.1159/isbn.978-3-318-01439-6 · Cited by 5,583
Generalizing the Hough transform to detect arbitrary shapes
Ballard, D. H. (1981) · Pattern Recognition · doi:10.1016/0031-3203(81)90009-1 · Cited by 4,371 · cited 1×
“Circular template matching is used to extract the field of view and images are cropped to the square bounding box of this circular field of view [26].”
MethodCheung, N., Mitchell, P., & Wong, T. Y. (2010) · The Lancet · doi:10.1016/s0140-6736(09)62124-3 · Cited by 3,288
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620 · cited 1×
“The use of multi-scale patches could potentially be beneficial as usage of multi-scale patches has shown promising results in other applications [44].”
BackgroundEditorial: special issue on learning from imbalanced data sets
Chawla, N. V., Japkowicz, N., & Kotcz, A. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007733 · Cited by 2,063 · cited 1×
“Previous works have shown that adding more informative samples to the training set can improve the performance of the learner substantially [9], [10], [12]–[14], [45], [46].”
BackgroundThe Boosting Approach to Machine Learning: An Overview
Schapire, R. E. (2003) · Lecture Notes in Statistics · doi:10.1007/978-0-387-21579-2_9 · Cited by 2,003 · cited 4×
“In boosting techniques, an ensemble of learners is trained where each of the consecutive learners uses a fixed, more informative training set [9], [10].”
BackgroundInfluentialThe Unreasonable Effectiveness of Data
Halevy, A., Norvig, P., & Pereira, F. (2009) · IEEE Intelligent Systems · doi:10.1109/mis.2009.36 · Cited by 1,767 · cited 1×
“Increasing the amount of data with robust reference labels to train the CNNs may also help to further improve classification performance [48].”
BackgroundMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 3×
“A simplified version of the boosting strategy is a two-step approach in which misclassified samples of an initial model are used as the training set of a second, independent learner [6], [11].”
BackgroundMethodRetinal Imaging and Image Analysis
Abramoff, M. D., Garvin, M. K., & Sonka, M. (2010) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2010.2084567 · Cited by 1,439 · cited 3×
“As hemorrhages and micro-aneurysms are similar in characteristics and are only differentiable by their size and color on color fundus images, they can be easily confused [16].”
BackgroundSelective Sampling Using the Query by Committee Algorithm
Freund, Y., Seung, H. S., Shamir, E., & Tishby, N. (1997) · Machine Learning · doi:10.1023/a:1007330508534 · Cited by 1,097 · cited 2×
“To achieve this, the training set is dynamically updated during the learning process of a single learner, putting more emphasis on informative samples [12]–[14].”
BackgroundMulti-column deep neural network for traffic sign classification
Cireşan, D., Meier, U., Masci, J., & Schmidhuber, J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.023 · Cited by 1,057
Roulette-wheel selection via stochastic acceptance
Lipowski, A., & Lipowska, D. (2012) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2011.12.004 · Cited by 761 · cited 1×
“in is selected randomly while in is selected with probability [32]–[34]:”
BackgroundAutomated detection of diabetic retinopathy on digital fundus images
Sinthanayothin, C., Boyce, J. F., Williamson, T. H., Cook, H. L., Mensah, E., Lal, S., & Usher, D. (2002) · Diabetic Medicine · doi:10.1046/j.1464-5491.2002.00613.x · Cited by 509 · cited 1×
“classifiers [20]–[23] were used for the detection of hemorrhages”
MethodAutomatic detection of red lesions in digital color fundus photographs
Niemeijer, M., van Ginneken, B., Staal, J., Suttorp-Schulten, M. S. A., & Abramoff, M. D. (2005) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2005.843738 · Cited by 504
Cruz-Roa, A. A., Arevalo Ovalle, J. E., Madabhushi, A., & González Osorio, F. A. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_50 · Cited by 403 · cited 1×
“Recent works on cancer detection and brain segmentation have shown CNN achieved remarkable performance [6]–[8].”
BackgroundLearning on the border: active learning in imbalanced data classification
Ertekin, S., Huang, J., Bottou, L., & Giles, L. (2007) · Proceedings of the sixteenth ACM conference on Conference on information and knowledge management · doi:10.1145/1321440.1321461 · Cited by 365 · cited 2×
“To achieve this, the training set is dynamically updated during the learning process of a single learner, putting more emphasis on informative samples [12]–[14].”
BackgroundOptimal Wavelet Transform for the Detection of Microaneurysms in Retina Photographs
Quellec, G., Lamard, M., Josselin, P. M., Cazuguel, G., Cochener, B., & Roux, C. (2008) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2008.920619 · Cited by 343 · cited 1×
“based onmorphological operations [18],wavelet operations [19]”
Splat Feature Classification With Application to Retinal Hemorrhage Detection in Fundus Images
Tang, L., Niemeijer, M., Reinhardt, J. M., Garvin, M. K., & Abramoff, M. D. (2013) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2012.2227119 · Cited by 185 · cited 1×
“separately on color fundus images [24], [25].”
Dynamic Sampling Approach to Training Neural Networks for Multiclass Imbalance Classification
Lin, M., Tang, K., & Yao, X. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2228231 · Cited by 165 · cited 4×
“Similar to our approach, a previous work has used a dynamic sampling approach to train a multi-layer perceptron (MLP) [13].”
BackgroundMethodInfluentialAutomated localization of breast cancer in DCE-MRI
Gubern‐Mérida, A., Martí, R., Melendez, J., Hauth, J., Mann, R. M., Karssemeijer, N., & Platel, B. (2014) · Medical Image Analysis · doi:10.1016/j.media.2014.12.001 · Cited by 149 · cited 1×
“In here, only false positives encountered on negative images were taken into account to prevent ambiguities in the reference annotation to influence the result [37], [38].”
BackgroundComputer-aided Detection versus Independent Double Reading of Masses on Mammograms
Karssemeijer, N., Otten, J. D. M., Verbeek, A. L. M., Groenewoud, J. H., de Koning, H. J., Hendriks, J. H. C. L., & Holland, R. (2003) · Radiology · doi:10.1148/radiol.2271011962 · Cited by 127 · cited 1×
“In here, only false positives encountered on negative images were taken into account to prevent ambiguities in the reference annotation to influence the result [37], [38].”
BackgroundPagh, R. (2014) · Springer eBooks · doi:10.1007/978-3-642-27848-8 · Cited by 101
Akram, U., & Khan, S. A. (2011) · Journal of Medical Systems · doi:10.1007/s10916-011-9802-2 · Cited by 91 · cited 1×
“classifiers [20]–[23] were used for the detection of hemorrhages”
MethodComparing Image Detection Algorithms Using Resampling
Samuelson, F. W., & Petrick, N. (2006) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2006.1625167 · Cited by 88 · cited 1×
“Bootstrap analysis with 10,000 bootstraps was used to compute 95% confidence intervals for the Az values [39], [40].”
MethodAutomatic Detection of Microaneurysms and Hemorrhages in Digital Fundus Images
Kande, G. B., Savithri, T. S., & Subbaiah, P. V. (2009) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-009-9246-0 · Cited by 86 · cited 1×
“based onmorphological operations [18],wavelet operations [19]”
Rasta, S. H., Partovi, M., Seyedarabi, H., & Javadzadeh, A. (2015) · Journal of Medical Signals & Sensors · doi:10.4103/2228-7477.150414 · Cited by 65 · cited 1×
“dimension to reduce the computational costs and preprocessing was applied to improve image contrast [27], [28].”
2D View Aggregation for Lymph Node Detection Using a Shallow Hierarchy of Linear Classifiers
Seff, A., Lü, L., Cherry, K. M., Roth, H. R., Liu, J., Wang, S., Hoffman, J., Turkbey, E. B., & Summers, R. M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10404-1_68 · Cited by 61
Automatic detection of cerebral microbleeds via deep learning based 3D feature representation
Chen, H., Yu, L., Dou, Q., Shi, L., Mok, V., & Heng, P. (2015) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2015.7163984 · Cited by 56
A survey on hemorrhage detection in diabetic retinopathy retinal images
Jitpakdee, P., Aimmanee, P., & Uyyanonvara, B. (2012) · International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON) · doi:10.1109/ecticon.2012.6254356 · Cited by 56 · cited 1×
“larger hemorrhages has received less attention [16], [17].”
BackgroundSegmenting Hippocampus from Infant Brains by Sparse Patch Matching with Deep-Learned Features
Guo, Y., Guorong Wu, A., Commander, L. A., Szary, S., Jewells, V. L., Lin, W., & Shen, D. G. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10470-6_39 · Cited by 53
van Grinsven, M. J. J. P., Lechanteur, Y. T. E., van de Ven, J. P. H., van Ginneken, B., Hoyng, C. B., Theelen, T., & Sánchez, C. I. (2013) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.12-11449 · Cited by 52 · cited 1×
“The local maxima locations are used as seed points for dynamic programming to segment the individual hemorrhage candidates [36].”
MethodSelf-citationWeighted Random Sampling: 2005; Efraimidis, Spirakis
Efraimidis, P., & Spirakis, P. (2008) · Encyclopedia of Algorithms · doi:10.1007/978-0-387-30162-4_478 · Cited by 45 · cited 1×
“in is selected randomly while in is selected with probability [32]–[34]:”
BackgroundImproving microaneurysm detection in color fundus images by using context-aware approaches
Antal, B., & Hajdú, A. (2013) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2013.05.001 · Cited by 37
Retinal image analysis: preprocessing and feature extraction
Marrugo, A. G., & Millán, M. S. (2011) · Journal of Physics Conference Series · doi:10.1088/1742-6596/274/1/012039 · Cited by 35 · cited 1×
“dimension to reduce the computational costs and preprocessing was applied to improve image contrast [27], [28].”
ADVANTAGES AND EXAMPLES OF RESAMPLING FOR CAD EVALUATION
Samuelson, F. W., Petrick, N., & Paquerault, S. (2007) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2007.356896 · Cited by 34 · cited 1×
“Bootstrap analysis with 10,000 bootstraps was used to compute 95% confidence intervals for the Az values [39], [40].”
MethodROC, LROC, FROC, AFROC: An Alphabet Soup
He, X. J., & Frey, E. C. (2009) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2009.06.001 · Cited by 29 · cited 2×
“One way to measure performance is by measuring the pixel classification performance using the area (Az) under the Receiver Operating Characteristics (ROC) curve [35].”
BackgroundMethodAn efficient parallel algorithm for random sampling
Rajan, V., Ghosh, R. K., & Gupta, P. (1989) · Information Processing Letters · doi:10.1016/0020-0190(89)90206-8 · Cited by 13
Certain Investigation of the Retinal Hemorrhage Detection in Fundus Images
Deepa, S., & Vijayprasath, S. (2015) · International Journal of Electronics and Communication Engineering · doi:10.14445/23488549/ijece-v2i2p106 · Cited by 4 · cited 1×
“separately on color fundus images [24], [25].”
Cited by
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
“Some researchers have specifically looked into tackling this imbalance by incorporating intelligence in the training process itself, by applying selective sampling (van Grinsven et al., 2016) or hard negative mining (Wang et al., 2016b).”
BackgroundSelf-citationDeep Learning for Health Informatics
Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000
“Example applications of CNNs in medical imaging include the classification of interstitial lung diseases based on computed tomography (CT) images [70], the classification of tuberculosis manifestation based on X-ray images [71], the classification of neural progenitor cells from somatic cell source [57], the detection…”
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Greenspan, H., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,700
“[24] attempt to improve and speed-up the CNN training for medical image analysis tasks by dynamically selecting misclassified negative samples during training.”
BackgroundBig data and machine learning algorithms for health-care delivery
Ngiam, K. Y., & Khor, I. W. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30149-4 · Cited by 1,424
Metadata
- Authors
- van Grinsven, M. J. J. P., van Ginneken, B., Hoyng, C. B., Theelen, T., & Sanchez, C. I.
- Affiliations
- Radboud University Medical Center (NL)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1273-1284
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2344912502
- Cited by
- 426
- References
- 51
- Citations by year
- 2026: 7 · 2025: 13 · 2024: 27 · 2023: 41 · 2022: 49 · 2021: 70 · 2020: 57 · 2019: 68 · 2018: 50 · 2017: 38 · 2016: 6
- FWCI
- 46.88
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- convolutional neural network training, training efficiency, sample weighting, hemorrhage detection, color fundus images, medical image analysis, training time reduction
- MeSH terms
- Machine Learning, Fundus Oculi, Humans, Image Interpretation, Computer-Assisted, Retinal Hemorrhage, Databases, Factual, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.