The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
The Power of Ensembles for Active Learning in Image Classification
Abstract
Deep learning methods have become the de-facto standard for challenging image processing tasks such as image classification. One major hurdle of deep learning approaches is that large sets of labeled data are necessary, which can be prohibitively costly to obtain, particularly in medical image diagnosis applications. Active learning techniques can alleviate this labeling effort. In this paper we investigate some recently proposed methods for active learning with high-dimensional data and convolutional neural network classifiers. We compare ensemble-based methods against Monte-Carlo Dropout and geometric approaches. We find that ensembles perform better and lead to more calibrated predictive uncertainties, which are the basis for many active learning algorithms. To investigate why Monte-Carlo Dropout uncertainties perform worse, we explore potential differences in isolation in a series of experiments. We show results for MNIST and CIFAR-10, on which we achieve a test set accuracy of 90% with roughly 12,200 labeled images, and initial results on ImageNet. Additionally, we show results on a large, highly class-imbalanced diabetic retinopathy dataset. We observe that the ensemble-based active learning effectively counteracts this imbalance during acquisition.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We observe that this difference is maintained in the other cycles, including the last active learning cycle where our method outperforms the semisupervised method, multi-model methods [9, 12] and the best AL method [5] by 1 . 6% .”
From Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection (2021) · cites this paper as [9]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
Hansen, L. K., & Salamon, P. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.58871 · Cited by 4,260
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917
(2013) · Cited by 2,056
Active Learning with Statistical Models
Cohn, D. A., Zoubin Ghahramani, A., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 2,054
Causes of vision loss worldwide, 1990–2010: a systematic analysis
Bourne, R., Stevens, G. A., White, R. A., Smith, J. L., Flaxman, S. R., Price, H., Jonas, J. B., Keeffe, J. E., Leasher, J. L., Naidoo, K. S., et al. (2013) · The Lancet Global Health · doi:10.1016/s2214-109x(13)70113-x · Cited by 1,882
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342
Weight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,335
Practical Variational Inference for Neural Networks
Graves, A. (2011) · Cited by 1,079
Diversity creation methods: a survey and categorisation
Gavin Brown, B., Wyatt, J., Harris, R., & Yao, X. (2004) · Information Fusion · doi:10.1016/j.inffus.2004.04.004 · Cited by 1,022
Deep Bayesian Active Learning with Image Data
Gal, Y., Islam, R., & Ghahramani, Z. (2017) · arXiv · doi:10.48550/arxiv.1703.02910 · Cited by 900
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837
Snapshot Ensembles: Train 1, get M for freePreprint
Huang, G. M., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00109 · Cited by 816
Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational InferencePreprint
Gal, Y., & Ghahramani, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02158 · Cited by 777
Estimating the mean and variance of the target probability distribution
Nix, D. A., & Weigend, A. S. (1994) · doi:10.1109/icnn.1994.374138 · Cited by 720
Cost-Effective Active Learning for Deep Image Classification
Wang, K., Zhang, D., Li, Y., Zhang, R., & Lin, L. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2589879 · Cited by 694
Active learning using pre-clustering
Nguyen, H. T., & Smeulders, A. W. M. (2004) · doi:10.1145/1015330.1015349 · Cited by 671
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625 · cited 1×
“Uncertainty calibration To assess calibration ([7]) quality we determine whether the expected fraction of correct classifications (as predicted by the model confidence, i.”
BackgroundMulti-class active learning for image classification
Joshi, A. J., Porikli, F., & Papanikolopoulos, N. P. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206627 · Cited by 588
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural NetworksPreprint
Hernández-Lobato, J. M., & Adams, R. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05336 · Cited by 563
Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation
Yang, L., Zhang, Y., Chen, J., Zhang, S., & Chen, D. Z. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_46 · Cited by 526
Bayesian Active Learning for Classification and Preference Learning
Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011) · arXiv · doi:10.48550/arxiv.1112.5745 · Cited by 507 · cited 1×
“Another measure (also known as (BALD, [27]) is the mutual information between data-points and weights.”
BackgroundCombining active learning and semi-supervised learning using Gaussian fields and harmonic functions
Zhu, X., Lafferty, J., & Ghahramani, Z. (2003) · Cited by 469
Multi-Class Active Learning by Uncertainty Sampling with Diversity Maximization
Yang, Y., Ma, Z., Nie, F., Chang, X., & Hauptmann, A. G. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0781-x · Cited by 448
Batch mode active learning and its application to medical image classification
Hoi, S. C. H., Jin, R., Zhu, J., & Lyu, M. R.-T. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143897 · Cited by 429
Resnikoff, S., Felch, W., Gauthier, T.-M., & Spivey, B. (2012) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2011-301378 · Cited by 405
Deep Bayesian Active Learning with Image Data
Ghahramani, Z., Gal, Y., & Islam, R. (2017) · PMLR · doi:10.17863/cam.11070 · Cited by 392
Shake-Shake regularizationPreprint
Gastaldi, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07485 · Cited by 353
Graham, B. (2014) · arXiv · doi:10.48550/arxiv.1412.6071 · Cited by 337
Active Learning with Gaussian Processes for Object Categorization
Kapoor, A., Grauman, K. L., Urtasun, R., & Darrell, T. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408844 · Cited by 334
Adaptive Active Learning for Image Classification
Li, X., & Guo, Y. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.116 · Cited by 279
Entropy-based active learning for object recognition
Holub, A. D., Perona, P., & Burl, M. C. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4563068 · Cited by 262
Stochastic Gradient VB and the Variational Auto-EncoderPreprint
Kingma, D. P., & Welling, M. (2013) · arXiv (Cornell University) · Cited by 234
Active learning: theory and applications
Koller, D., & Tong, S. (2001) · Cited by 182
Active Learning for Regression Based on Query by Committee
Burbidge, R., Rowland, J. J., & King, R. D. (2007) · Springer eBooks · doi:10.1007/978-3-540-77226-2_22 · Cited by 168
Scalable Active Learning for Multiclass Image Classification
Joshi, A. J., Porikli, F., & Papanikolopoulos, N. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.21 · Cited by 131
Elementary Applied Statistics for students in Behavioral Science
Freeman, L. G. (1965) · Revista Mexicana de Sociología · doi:10.2307/3538646 · Cited by 114
Iglesias, J. E., Konukoğlu, E., Montillo, A., Tu, Z., & Criminisi, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22092-0_3 · Cited by 66
Active and Continuous Exploration with Deep Neural Networks and Expected Model Output ChangesPreprint
Käding, C., Rodner, E., Freytag, A., & Denzler, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06129 · Cited by 58
The Well-Calibrated Bayesian: Rejoinder
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.2307/2287723 · Cited by 30
A Geometric Approach to Active Learning for Convolutional Neural NetworksPreprint
Şener, O., & Savarese, S. (2017) · arXiv (Cornell University) · Cited by 27
Uncertainty Decomposition in Bayesian Neural Networks with Latent VariablesPreprint
Depeweg, S., Hernández-Lobato, J. M., Doshi‐Velez, F., & Udluft, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.08495 · Cited by 21
The Importance of Diversity in Neural Network Ensembles - An Empirical Investigation
Johansson, U., Löfström, T., & Niklasson, L. (2007) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.2007.4371035 · Cited by 16
Efficient variational Bayesian neural network ensembles for outlier detectionPreprint
Pawlowski, N., Jaques, M., & Glocker, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06749 · Cited by 7
Diversity encouraging ensemble of convolutional networks for high performance action recognition
Yang, H., Yuan, C., Xing, J., & Hu, W. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296802 · Cited by 6
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“As additional measures to assess uncertainty quality, we report the negative log likelihood (NLL) and the Brier score ([3]) (as used in [40]) in Table 3 for four acquisition steps, and in Figure A7 across the whole acquisition procedure.”
Method
Research works printed but not linked: 3
- joshi, 2009, Multiclass active learning for image classification, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2372
- he, 2015, Deep residual learning for image recognition
- le cun, 1998, The MNIST Database of Handwritten Digits
Cited by
Sorted by citations
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
“• The results of our research show that although a variety of ensemble methods [474], [475], [476], [477], [478], [479], [480] also have a high ability to deal with uncertainty along with good performance and optimizing the performance of other methods, these high capabilities of these methods have not been used more…”
BackgroundEnsemble deep learning: A review
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105151 · Cited by 2,446
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,670
Uninformed Students: Student-Teacher Anomaly Detection With Discriminative Latent Embeddings
Bergmann, P., Fauser, M., Sattlegger, D., & Steger, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00424 · Cited by 990
Learning Loss for Active Learning
Yoo, D., & Kweon, I. S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00018 · Cited by 740
Metadata
- Authors
- Beluch, W. H., Genewein, T., Nurnberger, A., & Kohler, J. M.
- Affiliations
- Robert Bosch (United States) (US)
- ORCID
- Genewein, Tim
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 9368-9377
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2798820905
- Cited by
- 655
- References
- 64
- Citations by year
- 2026: 33 · 2025: 69 · 2024: 117 · 2023: 120 · 2022: 94 · 2021: 115 · 2020: 72 · 2019: 31 · 2018: 4
- FWCI
- 22.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- active learning (machine learning), image classification, ensemble methods, Monte Carlo dropout, uncertainty estimation, convolutional neural networks, class imbalance, diabetic retinopathy, medical image diagnosis, high-dimensional data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
