The Power of Ensembles for Active Learning in Image Classification
Abstract
Deep learning methods have become the de-facto standard for challenging image processing tasks such as image classification. One major hurdle of deep learning approaches is that large sets of labeled data are necessary, which can be prohibitively costly to obtain, particularly in medical image diagnosis applications. Active learning techniques can alleviate this labeling effort. In this paper we investigate some recently proposed methods for active learning with high-dimensional data and convolutional neural network classifiers. We compare ensemble-based methods against Monte-Carlo Dropout and geometric approaches. We find that ensembles perform better and lead to more calibrated predictive uncertainties, which are the basis for many active learning algorithms. To investigate why Monte-Carlo Dropout uncertainties perform worse, we explore potential differences in isolation in a series of experiments. We show results for MNIST and CIFAR-10, on which we achieve a test set accuracy of 90% with roughly 12,200 labeled images, and initial results on ImageNet. Additionally, we show results on a large, highly class-imbalanced diabetic retinopathy dataset. We observe that the ensemble-based active learning effectively counteracts this imbalance during acquisition.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On MS-COCO last active learning cycle, our method outperforms the semi-supervised method, multi-model methods, and the best AL method by 1.6%.
“We observe that this difference is maintained in the other cycles, including the last active learning cycle where our method outperforms the semisupervised method, multi-model methods [9, 12] and the best AL method [5] by 1 . 6% .”
From Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection (2021) · cites this paper as [9]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408 · see citing works
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791 · see citing works
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · see citing works
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276 · see citing works
Hansen, L. K., & Salamon, P. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.58871 · Cited by 4,260 · see citing works
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · see citing works
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088 · see citing works
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917 · see citing works
(2013) · Cited by 2,056 · see citing works
Active Learning with Statistical Models
Cohn, D. A., Zoubin Ghahramani, A., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 2,054 · see citing works
Causes of vision loss worldwide, 1990–2010: a systematic analysis
Bourne, R., Stevens, G. A., White, R. A., Smith, J. L., Flaxman, S. R., Price, H., Jonas, J. B., Keeffe, J. E., Leasher, J. L., Naidoo, K. S., et al. (2013) · The Lancet Global Health · doi:10.1016/s2214-109x(13)70113-x · Cited by 1,834 · see citing works
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · see citing works
Weight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,280 · see citing works
Practical Variational Inference for Neural Networks
Graves, A. (2011) · Cited by 1,079 · see citing works
Diversity creation methods: a survey and categorisation
Gavin Brown, B., Wyatt, J., Harris, R., & Yao, X. (2004) · Information Fusion · doi:10.1016/j.inffus.2004.04.004 · Cited by 1,003 · see citing works
Deep Bayesian Active Learning with Image Data
Gal, Y., Islam, R., & Ghahramani, Z. (2017) · arXiv · doi:10.48550/arxiv.1703.02910 · Cited by 900 · see citing works
Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational InferencePreprint
Gal, Y., & Ghahramani, Z. (2015) · arXiv.org · doi:10.48550/arxiv.1506.02158 · Cited by 839 · see citing works
Deep Exploration via Bootstrapped DQNPreprint
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 837 · see citing works
Estimating the mean and variance of the target probability distribution
Nix, D. A., & Weigend, A. S. (1994) · doi:10.1109/icnn.1994.374138 · Cited by 720 · see citing works
Snapshot Ensembles: Train 1, get M for freePreprint
Huang, G. M., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00109 · Cited by 700 · see citing works
Active learning using pre-clustering
Nguyen, H. T., & Smeulders, A. W. M. (2004) · doi:10.1145/1015330.1015349 · Cited by 671 · see citing works
Cost-Effective Active Learning for Deep Image Classification
Wang, K., Zhang, D., Li, Y., Zhang, R., & Lin, L. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2589879 · Cited by 669 · see citing works
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625 · cited 1× · see citing works
“Uncertainty calibration To assess calibration ([7]) quality we determine whether the expected fraction of correct classifications (as predicted by the model confidence, i.”
BackgroundMulti-class active learning for image classification
Joshi, A. J., Porikli, F., & Papanikolopoulos, N. P. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206627 · Cited by 588 · see citing works
Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation
Yang, L., Zhang, Y., Chen, J., Zhang, S., & Chen, D. Z. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_46 · Cited by 526 · see citing works
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural NetworksPreprint
Hernández-Lobato, J. M., & Adams, R. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05336 · Cited by 514 · see citing works
Bayesian Active Learning for Classification and Preference Learning
Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011) · arXiv · doi:10.48550/arxiv.1112.5745 · Cited by 507 · cited 1× · see citing works
“Another measure (also known as (BALD, [27]) is the mutual information between data-points and weights.”
BackgroundCombining active learning and semi-supervised learning using Gaussian fields and harmonic functions
Zhu, X., Lafferty, J., & Ghahramani, Z. (2003) · Cited by 469 · see citing works
Multi-Class Active Learning by Uncertainty Sampling with Diversity Maximization
Yang, Y., Ma, Z., Nie, F., Chang, X., & Hauptmann, A. G. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0781-x · Cited by 448 · see citing works
Batch mode active learning and its application to medical image classification
Hoi, S. C. H., Jin, R., Zhu, J., & Lyu, M. R.-T. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143897 · Cited by 429 · see citing works
Resnikoff, S., Felch, W., Gauthier, T.-M., & Spivey, B. (2012) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2011-301378 · Cited by 405 · see citing works
Deep Bayesian Active Learning with Image Data
Ghahramani, Z., Gal, Y., & Islam, R. (2017) · PMLR · doi:10.17863/cam.11070 · Cited by 392 · see citing works
Shake-Shake regularizationPreprint
Gastaldi, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07485 · Cited by 353 · see citing works
Graham, B. (2014) · arXiv · doi:10.48550/arxiv.1412.6071 · Cited by 337 · see citing works
Active Learning with Gaussian Processes for Object Categorization
Kapoor, A., Grauman, K. L., Urtasun, R., & Darrell, T. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408844 · Cited by 331 · see citing works
Adaptive Active Learning for Image Classification
Li, X., & Guo, Y. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.116 · Cited by 279 · see citing works
Entropy-based active learning for object recognition
Holub, A. D., Perona, P., & Burl, M. C. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4563068 · Cited by 262 · see citing works
Stochastic Gradient VB and the Variational Auto-EncoderPreprint
Kingma, D. P., & Welling, M. (2013) · arXiv (Cornell University) · Cited by 223 · see citing works
Active learning: theory and applications
Koller, D., & Tong, S. (2001) · Cited by 182 · see citing works
Active Learning for Regression Based on Query by Committee
Burbidge, R., Rowland, J. J., & King, R. D. (2007) · Springer eBooks · doi:10.1007/978-3-540-77226-2_22 · Cited by 168 · see citing works
Scalable Active Learning for Multiclass Image Classification
Joshi, A. J., Porikli, F., & Papanikolopoulos, N. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.21 · Cited by 130 · see citing works
Elementary Applied Statistics for students in Behavioral Science
Freeman, L. G. (1965) · Revista Mexicana de Sociología · doi:10.2307/3538646 · Cited by 114 · see citing works
Iglesias, J. E., Konukoğlu, E., Montillo, A., Tu, Z., & Criminisi, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22092-0_3 · Cited by 66 · see citing works
Active and Continuous Exploration with Deep Neural Networks and Expected Model Output ChangesPreprint
Käding, C., Rodner, E., Freytag, A., & Denzler, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06129 · Cited by 40 · see citing works
The Well-Calibrated Bayesian: Rejoinder
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.2307/2287723 · Cited by 30 · see citing works
A Geometric Approach to Active Learning for Convolutional Neural NetworksPreprint
Şener, O., & Savarese, S. (2017) · arXiv (Cornell University) · Cited by 27 · see citing works
Uncertainty Decomposition in Bayesian Neural Networks with Latent VariablesPreprint
Depeweg, S., Hernández-Lobato, J. M., Doshi‐Velez, F., & Udluft, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.08495 · Cited by 21 · see citing works
The Importance of Diversity in Neural Network Ensembles - An Empirical Investigation
Johansson, U., Löfström, T., & Niklasson, L. (2007) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.2007.4371035 · Cited by 16 · see citing works
Efficient variational Bayesian neural network ensembles for outlier detectionPreprint
Pawlowski, N., Jaques, M., & Glocker, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06749 · Cited by 7 · see citing works
Diversity encouraging ensemble of convolutional networks for high performance action recognition
Yang, H., Yuan, C., Xing, J., & Hu, W. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296802 · Cited by 6 · see citing works
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“As additional measures to assess uncertainty quality, we report the negative log likelihood (NLL) and the Brier score ([3]) (as used in [40]) in Table 3 for four acquisition steps, and in Figure A7 across the whole acquisition procedure.”
Method
Flagged first, then by citations
1 of the 64 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
“• The results of our research show that although a variety of ensemble methods [474], [475], [476], [477], [478], [479], [480] also have a high ability to deal with uncertainty along with good performance and optimizing the performance of other methods, these high capabilities of these methods have not been used more…”
BackgroundEnsemble deep learning: A review
Ganaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105151 · Cited by 2,282 · see citing works
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,525 · see citing works
Uninformed Students: Student-Teacher Anomaly Detection With Discriminative Latent Embeddings
Bergmann, P., Fauser, M., Sattlegger, D., & Steger, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00424 · Cited by 990 · see citing works
Learning Loss for Active Learning
Yoo, D., & Kweon, I. S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00018 · Cited by 740 · see citing works
A survey on active learning and human-in-the-loop deep learning for medical image analysis
Budd, S., Robinson, E. C., & Kainz, B. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102062 · Cited by 606 · see citing works
Variational Adversarial Active Learning
Sinha, S., Ebrahimi, S., & Darrell, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00607 · Cited by 585 · see citing works
Knockoff Nets: Stealing Functionality of Black-Box Models
Orekondy, T., Schiele, B., & Fritz, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00509 · Cited by 549 · see citing works
Methods for Segmentation and Classification of Digital Microscopy Tissue Images
Vu, Q. D., Graham, S., Kurç, T., To, M. N. N., Shaban, M., Qaiser, T., Koohbanani, N. A., Khurram, S. A., Kalpathy–Cramer, J., Zhao, T., et al. (2019) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00053 · Cited by 341 · see citing works
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
Feng, D., Harakeh, A., Waslander, S. L., & Dietmayer, K. (2020) · IEEE transactions on intelligent transportation systems (Print) · doi:10.1109/tits.2021.3096854 · Cited by 328 · see citing works
A Review on Machine Learning Styles in Computer Vision—Techniques and Future Directions
Mahadevkar, S. V., Khemani, B., Patil, S., Kotecha, K. V., Vora, D. R., Abraham, A., & Gabralla, L. A. (2022) · IEEE Access · doi:10.1109/access.2022.3209825 · Cited by 304 · see citing works
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
White, C., Neiswanger, W., & Savani, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17233 · Cited by 287 · see citing works
Scalia, G., Grambow, C. A., Pernici, B., Li, Y., & Green, W. H. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00975 · Cited by 285 · see citing works
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.-F., Barriuso, A., Torralba, A., & Fidler, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01001 · Cited by 259 · see citing works
A Survey of Uncertainty in Deep Neural Networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03342 · Cited by 232 · see citing works
AI Applications through the Whole Life Cycle of Material Discovery
Li, J., Lim, K., Yang, H., Ren, Z., Raghavan, S., Chen, P., Buonassisi, T., & Wang, X. (2020) · Matter · doi:10.1016/j.matt.2020.06.011 · Cited by 193 · see citing works
A Survey on Active Deep Learning: From Model Driven to Data Driven
Liu, P., Wang, L., Ranjan, R. K., He, G., & Zhao, L. (2022) · ACM Computing Surveys · doi:10.1145/3510414 · Cited by 199 · see citing works
Consistency-Based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
Gao, M., Zhang, Z., Yu, G., Arık, S. Ö., Davis, L. S., & Pfister, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58607-2_30 · Cited by 180 · see citing works
Contextual Diversity for Active Learning
Agarwal, S., Arora, H., Anand, S., & Arora, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_9 · Cited by 176 · see citing works
Applications, Challenges, and Future Directions of Human-in-the-Loop Learning
Kumar, S., Datta, S., Singh, V., Datta, D., Singh, S. K., & Sharma, R. (2024) · IEEE Access · doi:10.1109/access.2024.3401547 · Cited by 194 · see citing works
Weakly supervised machine learning
Ren, Z., Wang, S., & Zhang, Y. (2023) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12216 · Cited by 174 · see citing works
Multiple Instance Active Learning for Object Detection
Yuan, T., Wan, F., Fu, M., Liu, J., Xu, S., Ji, X., & Ye, Q. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00529 · Cited by 155 · see citing works
Active Learning for Deep Visual Tracking
Yuan, D., Chang, X.-J., Liu, Q., Yang, Y., Wang, D., Shu, M.-L., He, Z.-Y., & Shi, G.-M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3266837 · Cited by 159 · see citing works
Amin, S. U., Hussain, A., Kim, B., & Seo, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120391 · Cited by 161 · see citing works
ViewAL: Active Learning With Viewpoint Entropy for Semantic Segmentation
Siddiqui, Y., Valentin, J., & Nießner, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00945 · Cited by 145 · see citing works
Mundt, M., Hong, Y., Pliushch, I., & Ramesh, V. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.014 · Cited by 148 · see citing works
Active discovery of organic semiconductors
Künkel, C., Margraf, J. T., Chen, K., Oberhofer, H., & Reuter, K. (2021) · Nature Communications · doi:10.1038/s41467-021-22611-4 · Cited by 138 · see citing works
Active Learning for Deep Object Detection via Probabilistic Modeling
Choi, J., Elezi, I., Lee, H.-J., Farabet, C., & Alvarez, J. M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01010 · Cited by 135 · see citing works
“For ensembles, we follow [2], building an ensemble of three independent models.”
BackgroundMethodInfluentialA Survey on Deep Active Learning: Recent Advances and New Frontiers
Li, D., Wang, Z. R., Chen, Y., Jiang, R., Ding, W., & Okumura, M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3396463 · Cited by 145 · see citing works
State-Relabeling Adversarial Active Learning
Zhang, B., Li, L., Yang, S., Wang, S., Zha, Z.-J., & Huang, Q. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00878 · Cited by 132 · see citing works
Active Learning by Feature Mixing
Parvaneh, A., Abbasnejad, E., Teney, D., Haffari, R., van den Hengel, A., & Shi, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01192 · Cited by 134 · see citing works
Sequential Graph Convolutional Network for Active Learning
Caramalau, R., Bhattarai, B., & Kim, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00946 · Cited by 133 · see citing works
A comprehensive survey on deep active learning in medical image analysis
Wang, H., Jin, Q., Li, S., Liu, S., Wang, M., & Song, Z. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103201 · Cited by 139 · see citing works
Deep Reinforcement Active Learning for Human-in-the-Loop Person Re-Identification
Liu, Z., Wang, J., Gong, S., Tao, D., & Lu, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00622 · Cited by 115 · see citing works
Modern Bayesian Experimental Design
Rainforth, T., Foster, A., Ivanova, D. R., & Smith, F. B. (2024) · Statistical Science · doi:10.1214/23-sts915 · Cited by 110 · see citing works
Diminishing Uncertainty Within the Training Pool: Active Learning for Medical Image Segmentation
Nath, V., Yang, D., Landman, B. A., Xu, D., & Roth, H. R. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3048055 · Cited by 110 · see citing works
Scalable Active Learning for Object Detection
Haussmann, E., Fenzi, M., Chitta, K., Ivanecky, J., Xu, H., Roy, D., Mittel, A., Koumchatzky, N., Farabet, C., & Alvarez, J. M. (2020) · 2020 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/iv47402.2020.9304793 · Cited by 110 · see citing works
Wang, X., Chen, H., Xiang, H., Lin, H., Lin, X., & Heng, P. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102010 · Cited by 106 · see citing works
On Supervised Class-Imbalanced Learning: An Updated Perspective and Some Key Challenges
Das, S., Mullick, S. S., & Zelinka, I. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3160658 · Cited by 103 · see citing works
Yang, K., Tang, X., Li, J., Wang, H., Zhong, G., Chen, J., & Cao, D. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3270887 · Cited by 103 · see citing works
Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint
Wu, J., Chen, J., & Huang, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00918 · Cited by 104 · see citing works
Zhao, Z., Zeng, Z., Xu, K., Chen, C., & Guan, C. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3052320 · Cited by 88 · see citing works
Biderman, D., Whiteway, M. R., Hurwitz, C., Greenspan, N., Lee, R. S., Vishnubhotla, A., Warren, R., Pedraja, F., Noone, D., Schartner, M. M., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02319-1 · Cited by 100 · see citing works
Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector
Feng, D., Wei, X., Rosenbaum, L., Maki, A., & Dietmayer, K. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8814236 · Cited by 85 · see citing works
Shi, W., Yan, G., Li, Y., Li, H., Liu, T., Sun, C., Wang, G., Zhang, Y., Zou, Y., & Wu, D. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117316 · Cited by 83 · see citing works
Enhanced transfer learning with data augmentation
Su, J., Yu, X., Wang, X., Wang, Z., & Chao, G. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107602 · Cited by 87 · see citing works
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
Li, D., Ling, H., Kim, S. W., Kreis, K., Fidler, S., & Torralba, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02064 · Cited by 83 · see citing works
Deep Active Learning: Unified and Principled Method for Query and TrainingPreprint
Shui, C., Zhou, F., Gagné, C., & Wang, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09162 · Cited by 80 · see citing works
Leveraging Heteroscedastic Aleatoric Uncertainties for Robust Real-Time LiDAR 3D Object Detection
Feng, D., Rosenbaum, L., Timm, F., & Dietmayer, K. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814046 · Cited by 79 · see citing works
Consistency-based Active Learning for Object Detection
Yu, W., Zhu, S., Yang, T., & Chen, C. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00440 · Cited by 78 · see citing works
Yue, J., Fang, L., Ghamisi, P., Xie, W., Li, J., Chanussot, J., & Plaza, A. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3161377 · Cited by 79 · see citing works
Active Learning for Imbalanced Datasets
Aggarwal, U., Popescu, A., & Hudelot, C. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093475 · Cited by 75 · see citing works
Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
Ash, J. T., Zhang, C., Krishnamurthy, A., Langford, J. C., & Agarwal, A. (2020) · International Conference on Learning Representations · Cited by 74 · see citing works
Zou, Q., & Chen, S. (2021) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2021.107800 · Cited by 66 · see citing works
Active and incremental learning for semantic ALS point cloud segmentation
Lin, Y., George Vosselman, A., Cao, Y., & Yang, M. Y. (2020) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2020.09.003 · Cited by 65 · see citing works
Active, continual fine tuning of convolutional neural networks for reducing annotation efforts
Zhou, Z., Shin, J. Y., Gurudu, S., Gotway, M. B., & Liang, J. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101997 · Cited by 64 · see citing works
Revisiting Superpixels for Active Learning in Semantic Segmentation with Realistic Annotation Costs
Cai, L.-L., Xu, X., Liew, J., & Foo, C. S. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.01084 · Cited by 83 · see citing works
Deep active learning for object detection
Li, Y., Fan, B., Zhang, W., Ding, W., & Yin, J. (2021) · Information Sciences · doi:10.1016/j.ins.2021.08.019 · Cited by 60 · see citing works
Cai, Y., Chen, H., Yang, X. Y., Zhou, Y. C., & Cheng, K.-T. T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102794 · Cited by 62 · see citing works
Multiple-boundary clustering and prioritization to promote neural network retraining
Shen, W., Li, Y., Chen, L., Han, Y., Zhou, Y., & Xu, B. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416621 · Cited by 59 · see citing works
DIAL: Deep Interactive and Active Learning for Semantic Segmentation in Remote Sensing
Lenczner, G., Chan-Hon-Tong, A., Saux, B. L., Luminari, N., & Besnerais, G. L. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3166551 · Cited by 59 · see citing works
Active Exploration of Multimodal Complementarity for Few-Shot Action Recognition
Wanyan, Y., Yang, X., Chen, C., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00628 · Cited by 58 · see citing works
MCDAL: Maximum Classifier Discrepancy for Active Learning
Cho, J.-W., Kim, D.-J., Jung, Y., & Kweon, I.-S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3152786 · Cited by 57 · see citing works
A Survey on Green Deep LearningPreprint
Xu, J., Zhou, W., Fu, Z., Zhou, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05193 · Cited by 84 · see citing works
Clare, M., Sonnewald, M., Lguensat, R., Deshayes, J., & Balaji, V. (2022) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2022ms003162 · Cited by 56 · see citing works
Active learning for medical image segmentation with stochastic batches
Gaillochet, M., Desrosiers, C., & Lombaert, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102958 · Cited by 57 · see citing works
Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision
Gudovskiy, D., Hodgkinson, A., Yamaguchi, T., & Tsukizawa, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00906 · Cited by 55 · see citing works
Zhang, Y., Liu, P., Chen, L., Xu, M., Guo, X., & Zhao, L. (2023) · International Journal of Digital Earth · doi:10.1080/17538947.2023.2230978 · Cited by 54 · see citing works
Semantic Segmentation with Active Semi-Supervised Learning
Rangnekar, A., Kanan, C., & Hoffman, M. J. (2022) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00591 · Cited by 44 · see citing works
Taxonomy of Machine Learning Safety: A Survey and Primer
Mohseni, S., Wang, H., Xiao, C., Yu, Z., Wang, Z., & Yadawa, J. (2023) · ACM Computing Surveys · doi:10.1145/3551385 · Cited by 53 · see citing works
Intelligent labeling based on fisher information for medical image segmentation using deep learning
Sourati, J., Gholipour, A., Dy, J. G., Tomas-Fernandez, X., Kurugol, S., & Warfield, S. K. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2907805 · Cited by 50 · see citing works
Rethinking deep active learning: Using unlabeled data at model training
Siméoni, O., Budnik, M., Avrithis, Y., & Gravier, G. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412716 · Cited by 51 · see citing works
Duong‐Trung, N., Born, S., Kim, J. W., Schermeyer, M., Paulick, K., Borisyak, M., Bournazou, M. N. C., Werner, T., Scholz, R., Schmidt-Thieme, L., et al. (2022) · Biochemical Engineering Journal · doi:10.1016/j.bej.2022.108764 · Cited by 52 · see citing works
Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles
Jain, S., Liu, G., Mueller, J. W., & Gifford, D. K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5849 · Cited by 8 · see citing works
DEAL: Difficulty-Aware Active Learning for Semantic Segmentation
Xie, S., Feng, Z., Chen, Y., Sun, S., Ma, C., & Song, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69525-5_40 · Cited by 49 · see citing works
Deep Active Learning for Computer Vision Tasks: Methodologies, Applications, and Challenges
Wu, M., Li, C., & Yao, Z. (2022) · Applied Sciences · doi:10.3390/app12168103 · Cited by 51 · see citing works
Deep active learning models for imbalanced image classification
Jin, Q., Yuan, M., Wang, H., Wang, M., & Song, Z. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109817 · Cited by 47 · see citing works
Unlabeled data selection for active learning in image classification
Li, X., Wang, X., Chen, X., Lu, Y., Fu, H., & Wu, Y. C. (2024) · Scientific Reports · doi:10.1038/s41598-023-50598-z · Cited by 47 · see citing works
Sampling Bias in Deep Active Classification: An Empirical Study
Prabhu, A., Dognin, C., & Singh, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1417 · Cited by 46 · see citing works
All you need are a few pixels: semantic segmentation with PixelPick
Shin, G., Xie, W., & Albanie, S. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00194 · Cited by 46 · see citing works
Towards Robust and Reproducible Active Learning using Neural Networks
Munjal, P., Hayat, N., Hayat, M., Sourati, J., & Khan, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00032 · Cited by 46 · see citing works
VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning
Choi, J., Yi, K. M., Kim, J., Choo, J., Kim, B., Chang, J.-Y., Gwon, Y., & Chang, H. (2020) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00668 · Cited by 55 · see citing works
Ensemble Framework of Deep CNNs for Diabetic Retinopathy Detection
Gao, J., Qummar, S., Zhang, J., Yao, R., & Khan, F. G. (2020) · Computational Intelligence and Neuroscience · doi:10.1155/2020/8864698 · Cited by 46 · see citing works
Active Learning With Co-Auxiliary Learning and Multi-Level Diversity for Image Classification
Wang, Z., Chen, Z., & Du, B. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3234993 · Cited by 43 · see citing works
Sheikh, R., Milioto, A., Lottes, P., Stachniss, C., Bennewitz, M., & Schultz, T. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196722 · Cited by 43 · see citing works
Venugopalan, M., & Gupta, D. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118228 · Cited by 42 · see citing works
Wang, K., Wang, Y., Liu, B., & Chen, J. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3256459 · Cited by 42 · see citing works
Reducing DNN labelling cost using surprise adequacy: an industrial case study for autonomous driving
Kim, J.-H., Ju, J., FELDT, R. H., & Yoo, S. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417065 · Cited by 42 · see citing works
Image preprocessing‐based ensemble deep learning classification of diabetic retinopathy
Mácsik, P., Pavlovičová, J., Kajan, S., Goga, J., & Kurilová, V. (2023) · IET Image Processing · doi:10.1049/ipr2.12987 · Cited by 39 · see citing works
TRADI: Tracking Deep Neural Network Weight Distributions
Franchi, G., Bursuc, A., Aldea, E., Dubuisson, S., & Isabelle Bloch, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_7 · Cited by 39 · see citing works
Class-Balanced Active Learning for Image Classification
Bengar, J. Z., van de Weijer, J., Lopez-Fuentes, L., & Raducanu, B. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00376 · Cited by 39 · see citing works
Cold-start active learning for image classification
Jin, Q., Yuan, M., Li, S., Wang, H., Wang, M., & Song, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.066 · Cited by 37 · see citing works
A Representation-Based Query Strategy to Derive Qualitative Features for Improved Churn Prediction
De, S., & Prabu, P. (2023) · IEEE Access · doi:10.1109/access.2022.3233768 · Cited by 37 · see citing works
Accelerating high-entropy alloy discovery: efficient exploration via active learning
Sulley, G. A., Raush, J. R., Montemore, M. M., & Hamm, J. (2024) · Scripta Materialia · doi:10.1016/j.scriptamat.2024.116180 · Cited by 37 · see citing works
Shaikh, T. A., Rasool, T., Verma, P., & Mir, W. A. (2024) · Annals of Operations Research · doi:10.1007/s10479-024-06444-0 · Cited by 48 · see citing works
Mahanta, L. B., Hussain, E., Das, N., Kakoti, L., & Chowdhury, M. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107136 · Cited by 37 · see citing works
On Statistical Bias In Active Learning: How and When To Fix ItPreprint
Farquhar, S., Gal, Y., & Rainforth, T. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2101.11665 · Cited by 106 · see citing works
Counting People by Estimating People Flows
Liu, W., Salzmann, M., & Fua, P. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3102690 · Cited by 38 · see citing works
Ji, W., Liang, R., Zheng, Z., Zhang, W., Zhang, S., Li, J., Li, M., & Chua, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02204 · Cited by 35 · see citing works
Rethinking data collection for person re-identification: active redundancy reduction
Xu, X., Liu, L., Zhang, X., Guan, W., & Hu, R. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107827 · Cited by 34 · see citing works
- Authors
- Beluch, W. H., Genewein, T., Nurnberger, A., & Kohler, J. M.
- Affiliations
- Robert Bosch (United States) (US)
- ORCID
- Genewein, Tim
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 9368-9377
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2798820905
- Cited by
- 655
- References
- 64
- Citations by year
- 2026: 33 · 2025: 69 · 2024: 117 · 2023: 120 · 2022: 94 · 2021: 115 · 2020: 72 · 2019: 31 · 2018: 4
- FWCI
- 22.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- active learning (machine learning), image classification, ensemble methods, Monte Carlo dropout, uncertainty estimation, convolutional neural networks, class imbalance, diabetic retinopathy, medical image diagnosis, high-dimensional data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
