The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision
Open accessAbstract
While deep neural networks have become the go-to approach in computer vision, the vast majority of these models fail to properly capture the uncertainty inherent in their predictions. Estimating this predictive uncertainty can be crucial, for example in automotive applications. In Bayesian deep learning, predictive uncertainty is commonly decomposed into the distinct types of aleatoric and epistemic uncertainty. The former can be estimated by letting a neural network output the parameters of a certain probability distribution. Epistemic uncertainty estimation is a more challenging problem, and while different scalable methods recently have emerged, no extensive comparison has been performed in a real-world setting. We therefore accept this task and propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. Our proposed framework is specifically designed to test the robustness required in real-world computer vision applications. We also apply this framework to provide the first properly extensive and conclusive comparison of the two current state-of-the- art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates. Code is available at https://github.com/fregu856/evaluating_bdl.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Equation of State Calculations by Fast Computing Machines
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953) · The Journal of Chemical Physics · doi:10.1063/1.1699114 · Cited by 37,192
(2014) · Cited by 33,419
Monte Carlo sampling methods using Markov chains and their applications.
Hastings, W. K. (1970) · Biometrika · doi:10.1093/biomet/57.1.97 · Cited by 15,195
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
(2017) · Cited by 11,625
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655
MCMC Using Hamiltonian Dynamics
Neal, R. M. (2011) · Handbook of Markov Chain Monte Carlo · doi:10.1201/b10905-6 · Cited by 2,574
Bayesian Learning via Stochastic Gradient Langevin Dynamics
Welling, M., & Teh, Y. W. (2012) · Cited by 1,551
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342
Weight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,335
Huber, M. (2011) · Handbook of Markov Chain Monte Carlo · doi:10.1201/b10905-10 · Cited by 1,176
VirtualWorlds as Proxy for Multi-object Tracking Analysis
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.470 · Cited by 1,079
Practical Variational Inference for Neural Networks
Graves, A. (2011) · Cited by 1,078
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994
Obtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.57 · Cited by 784
The Loss Surfaces of Multilayer Networks
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.0233 · Cited by 733
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning
Yang, T.-J., Chen, Y.-H., & Sze, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.643 · Cited by 732
Evidential Deep Learning to Quantify Classification Uncertainty
Sensoy, M., Kaplan, L., & Kandemir, M. (2018) · arXiv · doi:10.48550/arxiv.1806.01768 · Cited by 694
Wang, Z., & Jia, K. (2019) · 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968513 · Cited by 534
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
Hernández-Lobato, J. M., & Adams, R. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05336 · Cited by 504
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera
Ma, F., Cavalheiro, G. V., & Karaman, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793637 · Cited by 460
OCNet: Object Context Network for Scene Parsing
Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X., & Wang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00916 · Cited by 453
Stochastic Gradient Hamiltonian Monte Carlo
Chen, T., Fox, E. B., & Guestrin, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.4102 · Cited by 434
Evaluating Bayesian Deep Learning Methods for Semantic SegmentationPreprint
Mukhoti, J., & Gal, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12709 · Cited by 272
Virtual Worlds as Proxy for Multi-Object Tracking Analysis
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06457 · Cited by 252
Synscapes: A Photorealistic Synthetic Dataset for Street Scene ParsingPreprint
Wrenninge, M., & Unger, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08705 · Cited by 243
Confidence Propagation through CNNs for Guided Sparse Depth Regression
Eldesokey, A., Felsberg, M., & Khan, F. S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929170 · Cited by 217
Uncertainty Estimates and Multi-hypotheses Networks for Optical Flow
Ilg, E., Çiçek, Ö., Galesso, S., Klein, A., Makansi, O., Hutter, F., & Brox, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_40 · Cited by 213
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
Louizos, C., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01961 · Cited by 210
Cyclical Stochastic Gradient MCMC for Bayesian Deep LearningPreprint
Zhang, R., Li, C., Zhang, J., Chen, C., & Wilson, A. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03932 · Cited by 186
Reliability, sufficiency, and the decomposition of proper scores
Bröcker, J. (2009) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.456 · Cited by 183
Lightweight Probabilistic Deep Networks
Gast, J., & Roth, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00355 · Cited by 169
Ensemble learning in Bayesian neural networks
Barber, D., & Bishop, C. (1998) · Cited by 166
Predictive Uncertainty Estimation via Prior Networks
Malinin, A., & Gales, M. (2018) · Apollo (University of Cambridge) · doi:10.17863/cam.35237 · Cited by 163
A complete recipe for stochastic gradient MCMC
Ma, Y.-A., Chen, T., & Fox, E. B. (2015) · neural information processing systems · Cited by 141
Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
Louizos, C., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1603.04733 · Cited by 138
Evaluating model calibration in classification
Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., & Schön, T. B. (2019) · arXiv · doi:10.48550/arxiv.1902.06977 · Cited by 117
Exponentially many local minima for single neurons
Auer, P. L., Herbster, M., & Warmuth, M. K. (1995) · Cited by 115
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2017) · Apollo (University of Cambridge) · doi:10.17863/cam.56535 · Cited by 114
Reliability, Sufficiency, and the Decomposition of Proper Scores
Bröcker, J. (2008) · MPG.PuRe (Max Planck Society) · Cited by 80
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
“[422] devised a comprehensive assessment framework for scalable epistemic uncertainty estimation techniques in deep learning.”
BackgroundLang, N., Kalischek, N., Armston, J., Schindler, K., Dubayah, R., & Wegner, J. D. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2021.112760 · Cited by 291
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., & Choi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.746 · Cited by 283
Scalia, G., Grambow, C. A., Pernici, B., Li, Y.-P., & Green, W. H. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00975 · Cited by 269
A Survey of Uncertainty in Deep Neural Networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03342 · Cited by 232
Metadata
- Authors
- Gustafsson, F. K., Danelljan, M., & Schon, T. B.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 1289-1298
- OpenAlex
- W2948210138
- Cited by
- 257
- References
- 57
- FWCI
- 17.03
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.