CoChat
Sign Up
PreprintCited by 371Adversarial Robustness in Machine LearningAdvanced Neural Network ApplicationsGaussian Processes and Bayesian Inference

The DOI matches its DataCite record; 97 of the 97 references it cites were found in a registry or an index.

A Survey of Uncertainty in Deep Neural Networks

Open access
Gawlikowski, Jakob, Tassi, Cedrique Rovile Njieutcheu, Ali, Mohsin, Lee, Jongseok, Humt, Matthias, Feng, Jianxiang, Kruspe, Anna, Triebel, Rudolph, Jung, Peter, Roscher, Ribana, Shahzad, Muhammad, Yang, Wen, Bamler, Richard, Zhu, Xiao Xiang·2021·arXiv
DOI 10.48550/arxiv.2107.03342arXiv 2107.03342
Open this paper in CoChat — free accountOpen PDF

Abstract

Due to their increasing spread, confidence in neural network predictions became more and more important. However, basic neural networks do not deliver certainty estimates or suffer from over or under confidence. Many researchers have been working on understanding and quantifying uncertainty in a neural network's prediction. As a result, different types and sources of uncertainty have been identified and a variety of approaches to measure and quantify uncertainty in neural networks have been proposed. This work gives a comprehensive overview of uncertainty estimation in neural networks, reviews recent advances in the field, highlights current challenges, and identifies potential research opportunities. It is intended to give anyone interested in uncertainty estimation in neural networks a broad overview and introduction, without presupposing prior knowledge in this field. A comprehensive introduction to the most crucial sources of uncertainty is given and their separation into reducible model uncertainty and not reducible data uncertainty is presented. The modeling of these uncertainties based on deterministic neural networks, Bayesian neural networks, ensemble of neural networks, and test-time data augmentation approaches is introduced and different branches of these fields as well as the latest developments are discussed. For a practical application, we discuss different measures of uncertainty, approaches for the calibration of neural networks and give an overview of existing baselines and implementations. Different examples from the wide spectrum of challenges in different fields give an idea of the needs and challenges regarding uncertainties in practical applications. Additionally, the practical limitations of current methods for mission- and safety-critical real world applications are discussed and an outlook on the next steps towards a broader usage of such methods is given.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.03342

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 240

  1. [3][3] P. Seeb¨ ock, J. I. Orlando, T. Schlegl, S. M. Waldstein, H. Bogunovi´ c, S. Klimscha, G. Langs, and U. Schmidt-Erfurth, 'Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal oct,' IEEE transactions on medical imaging , vol. 39, no. 1, pp. 87-98, 2019. -
  2. [11][11] K. Lee, H. Lee, K. Lee, and J. Shin, 'Training confidence-calibrated classifiers for detecting out-of-distribution samples,' in International Conference on Learning Representations , 2018. -
  3. [13][13] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek, 'Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift,' in Advances in Neural Information Processing Systems , 2019, pp. 13 99114 002. -
  4. [14][14] M. S. Ayhan and P. Berens, 'Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks,' in Medical Imaging with Deep Learning Conference , 2018. -
  5. [15][15] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, 'On calibration of modern neural networks,' in International Conference on Machine Learning . PMLR, 2017, pp. 1321-1330. -
  6. [16][16] A. G. Wilson and P. Izmailov, 'Bayesian deep learning and a probabilistic perspective of generalization,' in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 4697-4708. -
  7. [17][17] M. Rawat, M. Wistuba, and M.-I. Nicolae, 'Harnessing model uncertainty for detecting adversarial examples,' in NIPS Workshop on Bayesian Deep Learning , 2017. -
  8. [19][19] L. Smith and Y. Gal, 'Understanding measures of uncertainty for adversarial example detection,' in Proceesings of the Conference on Uncertainty in Artificial Intelligence , 2018, pp. 560-569. -
  9. [20][20] Y. Gal and Z. Ghahramani, 'Dropout as a bayesian approximation: Representing model uncertainty in deep learning,' in international conference on machine learning , 2016, pp. 1050-1059. -
  10. [22][22] J. Gawlikowski, S. Saha, A. Kruspe, and X. X. Zhu, 'Out-ofdistribution detection in satellite image classification,' in RobustML workshop at ICLR 2021 . ICRL, 2021, pp. 1-5. -
  11. [23][23] Y. Gal, R. Islam, and Z. Ghahramani, 'Deep bayesian active learning with image data,' in International Conference on Machine Learning . PMLR, 2017, pp. 1183-1192. -
  12. [29][29] B. L¨ otjens, M. Everett, and J. P. How, 'Safe reinforcement learning with model uncertainty estimates,' in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 8662-8668. -
  13. [30][30] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, 'Weight uncertainty in neural networks,' in Proceedings of the 32nd International Conference on International Conference on Machine LearningVolume 37 , 2015, pp. 1613-1622. -
  14. [31][31] B. Lakshminarayanan, A. Pritzel, and C. Blundell, 'Simple and scalable predictive uncertainty estimation using deep ensembles,' in Advances in neural information processing systems , 2017, pp. 6402-6413. -
  15. [32][32] A. Malinin and M. Gales, 'Predictive uncertainty estimation via prior networks,' in Advances in Neural Information Processing Systems , 2018, pp. 7047-7058. -
  16. [35][35] J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal, 'Uncertainty estimation using a single deep deterministic neural network,' in Proceedings of the 37th International Conference on Machine Learning . PMLR, 2020, pp. 9690-9700. -
  17. [39][39] M. Valdenegro-Toro, 'Deep sub-ensembles for fast uncertainty estimation in image classification,' in Bayesian Deep Learning Workshop at Neural Information Processing Systems 2019 , 2019. -
  18. [40][40] Y. Wen, D. Tran, and J. Ba, 'Batchensemble: an alternative approach to efficient ensemble and lifelong learning,' in 8th International Conference on Learning Representations , 2020. -
  19. [44][44] M. Sensoy, L. Kaplan, and M. Kandemir, 'Evidential deep learning to quantify classification uncertainty,' in Advances in Neural Information Processing Systems , 2018, pp. 3179-3189. -
  20. [45][45] A. Malinin, B. Mlodozeniec, and M. Gales, 'Ensemble distribution distillation,' in 8th International Conference on Learning Representations , 2020. -
  21. [46][46] M. Raghu, K. Blumer, R. Sayres, Z. Obermeyer, B. Kleinberg, S. Mullainathan, and J. Kleinberg, 'Direct uncertainty prediction for medical second opinions,' in International Conference on Machine Learning . PMLR, 2019, pp. 5281-5290. -
  22. [47][47] J. Wenger, H. Kjellstr¨ om, and R. Triebel, 'Non-parametric calibration for classification,' in International Conference on Artificial Intelligence and Statistics , 2020, pp. 178-190. -
  23. [48][48] J. Zhang, B. Kailkhura, and T. Y.-J. Han, 'Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning,' in International Conference on Machine Learning . PMLR, 2020, pp. 11 117-11 128. -
  24. [49][49] R. Ghanem, D. Higdon, and H. Owhadi, Handbook of uncertainty quantification . Springer, 2017, vol. 6. -
  25. [50][50] Y. Gal, 'Uncertainty in deep learning,' Ph.D. dissertation, University of Cambridge, 2016. -
  26. [51][51] A. G. Kendall, 'Geometry and uncertainty in deep learning for computer vision,' Ph.D. dissertation, University of Cambridge, 2019. -
  27. [52][52] A. Malinin, 'Uncertainty estimation in deep learning with application to spoken language assessment,' Ph.D. dissertation, University of Cambridge, 2019. -
  28. [54][54] --, 'A survey on bayesian deep learning,' ACM Computing Surveys (CSUR) , vol. 53, no. 5, pp. 1-37, 2020. -
  29. [55][55] N. St˚ ahl, G. Falkman, A. Karlsson, and G. Mathiason, 'Evaluation of uncertainty quantification in deep learning,' in Information Processing and Management of Uncertainty in Knowledge-Based Systems . Springer International Publishing, 2020, pp. 556-568. -
  30. [57][57] E. H¨ ullermeier and W. Waegeman, 'Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods,' Machine Learning , vol. 110, no. 3, pp. 457-506, 2021. -
  31. [60][60] A. Kendall and Y. Gal, 'What uncertainties do we need in bayesian deep learning for computer vision?' in Advances in neural information processing systems , 2017, pp. 5574-5584. -
  32. [63][63] H. Ritter, A. Botev, and D. Barber, 'A scalable laplace approximation for neural networks,' in 6th International Conference on Learning Representations , vol. 6. International Conference on Representation Learning, 2018. -
  33. [64][64] J. Nandy, W. Hsu, and M. L. Lee, 'Towards maximizing the representation gap between in-domain & out-of-distribution examples,' in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., 2020, pp. 9239-9250. -
  34. [65][65] A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov, 'Pitfalls of in-domain uncertainty estimation and ensembling in deep learning,' in International Conference on Learning Representations , 2020. -
  35. [67][67] D. Hendrycks and K. Gimpel, 'A baseline for detecting misclassified and out-of-distribution examples in neural networks,' in 5th International Conference on Learning Representations , 2017. -
  36. [68][68] S. Liang, Y. Li, and R. Srikant, 'Enhancing the reliability of out-ofdistribution image detection in neural networks,' in 6th International Conference on Learning Representations , 2018. -
  37. [69][69] A. Shafaei, M. Schmidt, and J. J. Little, 'A less biased evaluation of out-of-distribution sample detectors,' in British Machine Vision Conference 2019 , 2019. -
  38. [74][74] D. Barber and C. M. Bishop, 'Ensemble learning in bayesian neural networks,' Nato ASI Series F Computer and Systems Sciences , vol. 168, pp. 215-238, 1998. -
  39. [75][75] A. Graves, 'Practical variational inference for neural networks,' in Advances in neural information processing systems , 2011, pp. 23482356. -
  40. [76][76] C. Louizos, K. Ullrich, and M. Welling, 'Bayesian compression for deep learning,' in Advances in neural information processing systems , 2017, pp. 3288-3298. -
  41. [77][77] D. Rezende and S. Mohamed, 'Variational inference with normalizing flows,' in International Conference on Machine Learning , 2015, pp. 1530-1538. -
  42. [79][79] --, 'An improved acceptance procedure for the hybrid monte carlo algorithm,' Journal of Computational Physics , vol. 111, no. 1, pp. 194203, 1994. -
  43. [80][80] --, 'Bayesian learning for neural networks,' Ph.D. dissertation, University of Toronto, 1995. -
  44. [81][81] M. Welling and Y. W. Teh, 'Bayesian learning via stochastic gradient langevin dynamics,' in Proceedings of the 28th international conference on machine learning , 2011, pp. 681-688. -
  45. [83][83] T. Salimans and D. P. Kingma, 'Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks,' in Advances in Neural Information Processing Systems 29 . Curran Associates, Inc., 2016, pp. 901-909. -
  46. [84][84] J. Lee, M. Humt, J. Feng, and R. Triebel, 'Estimating model uncertainty of neural networks in sparse information form,' in International Conference on Machine Learning . PMLR, 2020, pp. 5702-5713. -
  47. [86][86] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, 'Snapshot ensembles: Train 1, get m for free,' in International conference on learning representations , 2017. -
  48. [87][87] G. D. Cavalcanti, L. S. Oliveira, T. J. Moura, and G. V. Carvalho, 'Combining diversity measures for ensemble pruning,' Pattern Recognition Letters , vol. 74, pp. 38-45, 2016. -
  49. [88][88] H. Guo, H. Liu, R. Li, C. Wu, Y. Guo, and M. Xu, 'Margin & diversity based ordering ensemble pruning,' Neurocomputing , vol. 275, pp. 237246, 2018. -
  50. [89][89] W. G. Martinez, 'Ensemble pruning via quadratic margin maximization,' IEEE Access , vol. 9, pp. 48 931-48 951, 2021. -
  51. [90][90] J. Lindqvist, A. Olmin, F. Lindsten, and L. Svensson, 'A general framework for ensemble distribution distillation,' in 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP) . IEEE, 2020, pp. 1-6. -
  52. [94][94] A. Malinin and M. Gales, 'Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness,' in Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d ´ Alch´ e-Buc, E. Fox, and R. Garnett, Eds., 2019, pp. 14 547-14 558. -
  53. [95][95] V. T. Vasudevan, A. Sethy, and A. R. Ghias, 'Towards better confidence estimation for neural models,' in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 7335-7339. -
  54. [96][96] M. Hein, M. Andriushchenko, and J. Bitterwolf, 'Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 41-50. -
  55. [97][97] T. Joo, U. Chung, and M.-G. Seo, 'Being bayesian about categorical probability,' in International Conference on Machine Learning . PMLR, 2020, pp. 4950-4961. -
  56. [99][99] --, 'Information robust dirichlet networks for predictive uncertainty estimation,' arXiv preprint arXiv:1910.04819 , 2019. - · arXiv
  57. [101][101] A. Amini, W. Schwarting, A. Soleimany, and D. Rus, 'Deep evidential regression,' in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 14 927-14 937. -
  58. [102][102] B. Charpentier, D. Z¨ ugner, and S. G¨ unnemann, 'Posterior network: Uncertainty estimation without ood samples via density-based pseudocounts,' in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 1356-1367. -
  59. [103][103] N. Tagasovska and D. Lopez-Paz, 'Single-model uncertainties for deep learning,' in Advances in Neural Information Processing Systems , 2019, pp. 6417-6428. -
  60. [106][106] J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield, 'Large automatic learning, rule extraction, and generalization,' Complex systems , vol. 1, no. 5, pp. 877-922, 1987. -
  61. [108][108] W. L. Buntine and A. S. Weigend, 'Bayesian back-propagation,' Complex systems , vol. 5, no. 6, pp. 603-643, 1991. -
  62. [109][109] D. J. C. MacKay, 'Bayesian model comparison and backprop nets,' in Advances in neural information processing systems , 1992, pp. 839-846. -
  63. [111][111] A. Corduneanu and C. M. Bishop, 'Variational bayesian model selection for mixture distributions,' in Artificial intelligence and Statistics , vol. 2001. Morgan Kaufmann Waltham, MA, 2001, pp. 27-34. -
  64. [112][112] S. Ghosh, J. Yao, and F. Doshi-Velez, 'Model selection in bayesian neural networks via horseshoe priors,' Journal of Machine Learning Research , vol. 20, no. 182, pp. 1-46, 2019. -
  65. [114][114] J. Achterhold, J. M. Koehler, A. Schmeink, and T. Genewein, 'Variational network quantization,' in International Conference on Learning Representations , 2018. -
  66. [116][116] A. Kirsch, J. van Amersfoort, and Y. Gal, 'Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning,' in Advances in Neural Information Processing Systems , 2019, pp. 70267037. -
  67. [117][117] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, 'Variational continual learning,' in International Conference on Learning Representations , 2018. -
  68. [118][118] S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach, 'Uncertaintyguided continual learning with bayesian neural networks,' in International Conference on Learning Representations , 2020. -
  69. [121][121] M. E. E. Khan, A. Immer, E. Abedi, and M. Korzepa, 'Approximate inference turns deep networks into gaussian processes,' in Advances in neural information processing systems , 2019, pp. 3094-3104. -
  70. [122][122] J. S. Denker and Y. LeCun, 'Transforming neural-net output levels to probability distributions,' in Advances in neural information processing systems , 1991, pp. 853-859. -
  71. [124][124] J. Hernandez-Lobato, Y. Li, M. Rowland, T. Bui, D. Hern´ andezLobato, and R. Turner, 'Black-box alpha divergence minimization,' in International Conference on Machine Learning , 2016, pp. 1511-1520. -
  72. [125][125] Y. Li and Y. Gal, 'Dropout inference in bayesian neural networks with alpha-divergences,' in International Conference on Machine Learning , 2017, pp. 2052-2061. -
  73. [127][127] T. P. Minka, 'Expectation propagation for approximate bayesian inference,' in Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence , 2001, pp. 362-369. -
  74. [129][129] J. M. Hern´ andez-Lobato and R. Adams, 'Probabilistic backpropagation for scalable learning of bayesian neural networks,' in International Conference on Machine Learning , 2015, pp. 1861-1869. -
  75. [131][131] D. Tran, M. D. Hoffman, R. A. Saurous, E. Brevdo, K. Murphy, and D. M. Blei, 'Deep probabilistic programming,' in International Conference on Machine Learning , 2016. -
  76. [132][132] E. Bingham, J. P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. Szerlip, P. Horsfall, and N. D. Goodman, 'Pyro: Deep universal probabilistic programming,' The Journal of Machine Learning Research , vol. 20, no. 1, pp. 973-978, 2019. -
  77. [133][133] R. Caba˜ nas, A. Salmer´ on, and A. R. Masegosa, 'Inferpy: Probabilistic modeling with tensorflow made easy,' Knowledge-Based Systems , vol. 168, pp. 25-27, 2019. -
  78. [135][135] S. Sun, G. Zhang, J. Shi, and R. Grosse, 'Functional variational bayesian neural networks,' in International Conference on Learning Representations , 2018. -
  79. [138][138] J. Postels, F. Ferroni, H. Coskun, N. Navab, and F. Tombari, 'Samplingfree epistemic uncertainty estimation using approximated variance propagation,' in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 2931-2940. -
  80. [140][140] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft, 'Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning,' in International Conference on Machine Learning . PMLR, 2018, pp. 1184-1193. -
  81. [141][141] --, 'Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning,' in Proceedings of the 35th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10-15 Jul 2018, pp. 1184-1193. -
  82. [142][142] D. P. Kingma, T. Salimans, and M. Welling, 'Variational dropout and the local reparameterization trick,' in Advances in neural information processing systems , 2015, pp. 2575-2583. -
  83. [143][143] A. Wu, S. Nowozin, E. Meeds, R. Turner, J. Hern´ andez-Lobato, and A. Gaunt, 'Deterministic variational inference for robust bayesian neural networks,' in 7th International Conference on Learning Representations, ICLR 2019 , 2019. -
  84. [144][144] C. Louizos and M. Welling, 'Structured and efficient variational deep learning with matrix gaussian posteriors,' in International Conference on Machine Learning , 2016, pp. 1708-1716. -
  85. [145][145] G. Zhang, S. Sun, D. Duvenaud, and R. Grosse, 'Noisy natural gradient as variational inference,' in International Conference on Machine Learning , 2018, pp. 5852-5861. -
  86. [146][146] S. Sun, C. Chen, and L. Carin, 'Learning structured weight uncertainty in bayesian neural networks,' in Artificial Intelligence and Statistics , 2017, pp. 1283-1292. -
  87. [148][148] A. Mishkin, F. Kunstner, D. Nielsen, M. Schmidt, and M. E. Khan, 'Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient,' in Advances in Neural Information Processing Systems , 2018, pp. 6245-6255. -
  88. [149][149] C. Louizos and M. Welling, 'Multiplicative normalizing flows for variational bayesian neural networks,' in International Conference on Machine Learning , 2017, pp. 2218-2227. -
  89. [150][150] K. Osawa, S. Swaroop, M. E. E. Khan, A. Jain, R. Eschenhagen, R. E. Turner, and R. Yokota, 'Practical deep learning with bayesian principles,' in Advances in neural information processing systems , 2019, pp. 4287-4299. -
  90. [151][151] Y. Gal, J. Hron, and A. Kendall, 'Concrete dropout,' in Advances in neural information processing systems , 2017, pp. 3581-3590. -
  91. [152][152] Z. Eaton-Rosen, F. Bragman, S. Bisdas, S. Ourselin, and M. J. Cardoso, 'Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions,' in International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2018, pp. 691-699. -
  92. [153][153] C. R. N. Tassi, 'Bayesian convolutional neural network: Robustly quantify uncertainty for misclassifications detection,' in Mediterranean Conference on Pattern Recognition and Artificial Intelligence . Springer, 2019, pp. 118-132. -
  93. [155][155] M. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava, 'Fast and scalable bayesian deep learning by weight-perturbation in adam,' in International Conference on Machine Learning . PMLR, 2018, pp. 2611-2620. -
  94. [156][156] M. E. Khan, Z. Liu, V. Tangkaratt, and Y. Gal, 'Vprop: Variational inference using rmsprop,' in Advances in neural information processing systems , 2017, pp. 3288-3298. -
  95. [157][157] A. Atanov, A. Ashukha, D. Molchanov, K. Neklyudov, and D. Vetrov, 'Uncertainty estimation via stochastic batch normalization,' in International Symposium on Neural Networks . Springer, 2019, pp. 261-269. -
  96. [158][158] S. Duane, A. D. Kennedy, B. J. Pendleton, and D. Roweth, 'Hybrid monte carlo,' Physics letters B , vol. 195, no. 2, pp. 216-222, 1987. -
  97. [160][160] K. A. Dubey, S. J. Reddi, S. A. Williamson, B. Poczos, A. J. Smola, and E. P. Xing, 'Variance reduction in stochastic gradient langevin dynamics,' in Advances in neural information processing systems , 2016, pp. 1154-1162. -
  98. [161][161] B. Leimkuhler and S. Reich, Simulating hamiltonian dynamics . Cambridge university press, 2004, vol. 14. -
  99. [162][162] P. J. Rossky, J. Doll, and H. Friedman, 'Brownian dynamics as smart monte carlo simulation,' The Journal of Chemical Physics , vol. 69, no. 10, pp. 4628-4633, 1978. -
  100. [163][163] G. O. Roberts and O. Stramer, 'Langevin diffusions and metropolishastings algorithms,' Methodology and computing in applied probability , vol. 4, no. 4, pp. 337-357, 2002. -
  101. [164][164] H. Kushner and G. G. Yin, Stochastic approximation and recursive algorithms and applications . Springer Science & Business Media, 2003, vol. 35. -
  102. [165][165] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning . MIT press Cambridge, 2016, vol. 1, no. 2. -
  103. [166][166] Y.-A. Ma, T. Chen, and E. Fox, 'A complete recipe for stochastic gradient mcmc,' in Advances in Neural Information Processing Systems , 2015, pp. 2917-2925. -
  104. [167][167] G. Marceau-Caron and Y. Ollivier, 'Natural langevin dynamics for neural networks,' in International Conference on Geometric Science of Information . Springer, 2017, pp. 451-459. -
  105. [168][168] Z. Nado, J. Snoek, R. B. Grosse, D. Duvenaud, B. Xu, and J. Martens, 'Stochastic gradient langevin dynamics that exploit neural network structure,' in International Conference on Learning Representations (Workshop) , 2018. -
  106. [169][169] U. Simsekli, R. Badeau, A. T. Cemgil, and G. Richard, 'Stochastic quasi-newton langevin monte carlo,' in Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48 , 2016, pp. 642-651. -
  107. [170][170] Y. Zhang and C. A. Sutton, 'Quasi-newton methods for markov chain monte carlo,' in Advances in Neural Information Processing Systems , 2011, pp. 2393-2401. -
  108. [171][171] T. Fu, L. Luo, and Z. Zhang, 'Quasi-newton hamiltonian monte carlo.' in Conference on Uncertainty in Artificial Intelligence , 2016. -
  109. [173][173] S. Ahn, A. K. Balan, and M. Welling, 'Bayesian posterior sampling via stochastic gradient fisher scoring,' in International Conference on Learning Representations , 2012. -
  110. [174][174] S. Patterson and Y. W. Teh, 'Stochastic gradient riemannian langevin dynamics on the probability simplex,' in Advances in neural information processing systems , 2013, pp. 3102-3110. -
  111. [176][176] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven, 'Bayesian sampling using stochastic gradient thermostats,' in Advances in neural information processing systems , 2014, pp. 3203-3211. -
  112. [177][177] X. Shang, Z. Zhu, B. Leimkuhler, and A. J. Storkey, 'Covariancecontrolled adaptive langevin thermostat for large-scale bayesian sampling,' in Advances in Neural Information Processing Systems , 2015, pp. 37-45. -
  113. [178][178] B. Leimkuhler and X. Shang, 'Adaptive thermostats for noisy gradient systems,' SIAM Journal on Scientific Computing , vol. 38, no. 2, pp. A712-A736, 2016. -
  114. [179][179] S. Ahn, B. Shahbaba, and M. Welling, 'Distributed stochastic gradient mcmc,' in International conference on machine learning , 2014, pp. 1044-1052. -
  115. [180][180] K.-C. Wang, P. Vicol, J. Lucas, L. Gu, R. Grosse, and R. Zemel, 'Adversarial distillation of bayesian neural network posteriors,' in International Conference on Machine Learning , 2018, pp. 5190-5199. -
  116. [181][181] A. K. Balan, V. Rathod, K. P. Murphy, and M. Welling, 'Bayesian dark knowledge,' in Advances in Neural Information Processing Systems , 2015, pp. 3438-3446. -
  117. [182][182] D. Zou, P. Xu, and Q. Gu, 'Stochastic variance-reduced hamilton monte carlo methods,' in International Conference on Machine Learning , 2018, pp. 6028-6037. -
  118. [183][183] A. Durmus, U. Simsekli, E. Moulines, R. Badeau, and G. Richard, 'Stochastic gradient richardson-romberg markov chain monte carlo,' in Advances in Neural Information Processing Systems , 2016, pp. 20472055. -
  119. [185][185] I. Sato and H. Nakagawa, 'Approximation analysis of stochastic gradient langevin dynamics by using fokker-planck equation and ito process,' in International Conference on Machine Learning , 2014, pp. 982-990. -
  120. [186][186] C. Chen, N. Ding, and L. Carin, 'On the convergence of stochastic gradient mcmc algorithms with high-order integrators,' in Advances in Neural Information Processing Systems , 2015, pp. 2278-2286. -
  121. [187][187] Y. W. Teh, A. H. Thiery, and S. J. Vollmer, 'Consistency and fluctuations for stochastic gradient langevin dynamics,' The Journal of Machine Learning Research , vol. 17, no. 1, pp. 193-225, 2016. -
  122. [189][189] F. Wenzel, K. Roth, B. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin, 'How good is the bayes posterior in deep neural networks really?' in International Conference on Machine Learning . PMLR, 2020, pp. 10 248-10 259. -
  123. [190][190] N. Ye, Z. Zhu, and R. K. Mantiuk, 'Langevin dynamics with continuous tempering for training deep neural networks,' in Proceedings of the 31st International Conference on Neural Information Processing Systems , 2017, pp. 618-626. -
  124. [191][191] R. Chandra, K. Jain, R. V. Deo, and S. Cripps, 'Langevin-gradient parallel tempering for bayesian neural learning,' Neurocomputing , vol. 359, pp. 315-326, 2019. -
  125. [192][192] A. Botev, H. Ritter, and D. Barber, 'Practical gauss-newton optimisation for deep learning,' in Proceedings of the 34th International Conference on Machine Learning-Volume 70 , 2017, pp. 557-565. -
  126. [193][193] J. Martens and R. Grosse, 'Optimizing neural networks with kroneckerfactored approximate curvature,' in Proceeding of the 32nd International conference on machine learning , 2015, pp. 2408-2417. -
  127. [194][194] S. Becker and Y. LeCun, 'Improving the convergence of backpropagation learning with second-order methods,' in Proceedings of the 1988 Connectionist Models Summer School, San Mateo , D. Touretzky, G. Hinton, and T. Sejnowski, Eds. Morgan Kaufmann, 1989, pp. 29-37. -
  128. [196][196] P. Hennig, 'Fast probabilistic optimization from noisy gradients,' in Proceedings of the 30th International Conference on Machine Learning , vol. 28-1. PMLR, 2013, pp. 62-70. -
  129. [197][197] N. L. Roux and A. W. Fitzgibbon, 'A fast natural newton method,' in Proceedings of the International Conference on Machine Learning , 2010. -
  130. [198][198] R. B. Grosse and J. Martens, 'A kronecker-factored approximate fisher matrix for convolution layers,' in Proceedings of the 33nd International Conference on Machine Learning , 2016, pp. 573-582. -
  131. [200][200] J. Ba, R. Grosse, and J. Martens, 'Distributed second-order optimization using kronecker-factored approximations,' in International Conference on Learning Representations , 2017. -
  132. [201][201] T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent, 'Fast approximate natural gradient descent in a kronecker factored eigenbasis,' in Advances in Neural Information Processing Systems , 2018, pp. 9573-9583. -
  133. [202][202] Y. LeCun, J. S. Denker, and S. A. Solla, 'Optimal brain damage,' in Advances in Neural Information Processing Systems 2 , 1990, pp. 598-605. -
  134. [203][203] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , 'Overcoming catastrophic forgetting in neural networks,' Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521-3526, 2017. -
  135. [204][204] A. Kristiadi, M. Hein, and P. Hennig, 'Being bayesian, even just a bit, fixes overconfidence in relu networks,' in International Conference on Machine Learning . PMLR, 2020, pp. 5436-5446. -
  136. [207][207] K. Shinde, J. Lee, M. Humt, A. Sezgin, and R. Triebel, 'Learning multiplicative interactions with bayesian neural networks for visualinertial odometry,' in Workshop on AI for Autonomous Driving at the 37th International Conference on Machine Learning , 2020. -
  137. [208][208] J. Feng, M. Durner, Z.-C. Marton, F. Balint-Benczedi, and R. Triebel, 'Introspective robot perception using smoothed predictions from bayesian neural networks,' in International Symposium on Robotics Research , 2019. -
  138. [210][210] A. Immer, M. Korzepa, and M. Bauer, 'Improving predictions of bayesian neural nets via local linearization,' in Proceedings of The 24th International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 703-711. -
  139. [213][213] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson, 'A simple baseline for bayesian uncertainty in deep learning,' in Advances in Neural Information Processing Systems , 2019, pp. 13 15313 164. -
  140. [217][217] O. Sagi and L. Rokach, 'Ensemble learning: A survey,' WIREs Data Mining and Knowledge Discovery , vol. 8, no. 4, p. e1249, 2018. -
  141. [218][218] L. K. Hansen and P. Salamon, 'Neural network ensembles,' IEEE transactions on pattern analysis and machine intelligence , vol. 12, no. 10, pp. 993-1001, 1990. -
  142. [219][219] Y. Cao, T. A. Geddes, J. Y. H. Yang, and P. Yang, 'Ensemble deep learning in bioinformatics,' Nature Machine Intelligence , pp. 1-9, 2020. -
  143. [221][221] L. Wei, S. Wan, J. Guo, and K. K. Wong, 'A novel hierarchical selective ensemble classifier with bioinformatics application,' Artificial intelligence in medicine , vol. 83, pp. 82-90, 2017. -
  144. [222][222] F. Lv, M. Han, and T. Qiu, 'Remote sensing image classification based on ensemble extreme learning machine with stacked autoencoder,' IEEE Access , vol. 5, pp. 9021-9031, 2017. -
  145. [223][223] X. Dai, X. Wu, B. Wang, and L. Zhang, 'Semisupervised scene classification for remote sensing images: A method based on convolutional neural networks and ensemble learning,' IEEE Geoscience and Remote Sensing Letters , vol. 16, no. 6, pp. 869-873, 2019. -
  146. [224][224] E. Marushko and A. Doudkin, 'Methods of using ensembles of heterogeneous models to identify remote sensing objects,' Pattern Recognition and Image Analysis , vol. 30, no. 2, pp. 211-216, 2020. -
  147. [225][225] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel, 'Modelensemble trust-region policy optimization,' in International Conference on Learning Representations , 2018. -
  148. [226][226] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine, 'Epopt: Learning robust neural network policies using model ensembles,' in International Conference on Learning Representations , 2017. -
  149. [228][228] A. Renda, M. Barsacchi, A. Bechini, and F. Marcelloni, 'Comparing ensemble strategies for deep learning: An application to facial expression recognition,' Expert Systems with Applications , vol. 136, pp. 1-11, 2019. -
  150. [229][229] E. J. Herron, S. R. Young, and T. E. Potok, 'Ensembles of networks produced from neural architecture search,' in International Conference on High Performance Computing . Springer, 2020, pp. 223-234. -
  151. [231][231] I. E. Livieris, L. Iliadis, and P. Pintelas, 'On ensemble techniques of weight-constrained neural networks,' Evolving Systems , pp. 1-13, 2020. -
  152. [232][232] L. Nanni, S. Brahnam, and G. Maguolo, 'Data augmentation for building an ensemble of convolutional neural networks,' in Innovation in Medicine and Healthcare Systems, and Multimedia . Singapore: Springer Singapore, 2019, pp. 61-69. -
  153. [234][234] R. Rahaman and A. H. Thiery, 'Uncertainty quantification and deep ensembles,' stat , vol. 1050, p. 20, 2020. -
  154. [235][235] Y. Wen, G. Jerfel, R. Muller, M. W. Dusenberry, J. Snoek, B. Lakshminarayanan, and D. Tran, 'Combining ensembles and data augmentation can harm your calibration,' in International Conference on Learning Representations , 2021. -
  155. [236][236] W. Kim, B. Goyal, K. Chawla, J. Lee, and K. Kwon, 'Attention-based ensemble for deep metric learning,' in Proceedings of the European Conference on Computer Vision , 2018. -
  156. [237][237] J. Yang and F. Wang, 'Auto-ensemble: An adaptive learning rate scheduling based deep learning model ensembling,' IEEE Access , vol. 8, pp. 217 499-217 509, 2020. -
  157. [238][238] M. Leutbecher and T. N. Palmer, 'Ensemble forecasting,' Journal of computational physics , vol. 227, no. 7, pp. 3515-3539, 2008. -
  158. [239][239] W. S. Parker, 'Ensemble modeling, uncertainty and robust predictions,' Wiley Interdisciplinary Reviews: Climate Change , vol. 4, no. 3, pp. 213-223, 2013. -
  159. [241][241] A. Vyas, N. Jammalamadaka, X. Zhu, D. Das, B. Kaul, and T. L. Willke, 'Out-of-distribution detection using an ensemble of self supervised leave-out classifiers,' in Proceedings of the European Conference on Computer Vision , 2018, pp. 550-564. -
  160. [242][242] J. Koci´ c, N. Joviˇ ci´ c, and V. Drndarevi´ c, 'An end-to-end deep neural network for autonomous driving designed for embedded automotive platforms,' Sensors , vol. 19, no. 9, p. 2064, 2019. -
  161. [243][243] G. Mart´ ınez-Mu˜ noz, D. Hern´ andez-Lobato, and A. Su´ arez, 'An analysis of ensemble pruning techniques based on ordered aggregation,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 31, no. 2, pp. 245-259, 2008. -
  162. [244][244] C. Buciluˇ a, R. Caruana, and A. Niculescu-Mizil, 'Model compression,' in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , 2006, pp. 535-541. -
  163. [245][245] G. Hinton, O. Vinyals, and J. Dean, 'Distilling the knowledge in a neural network,' stat , vol. 1050, p. 9, 2015. -
  164. [246][246] E. Englesson and H. Azizpour, 'Efficient evaluation-time uncertainty estimation by improved distillation,' in Workshop on Uncertainty and Robustness in Deep Learning at International Conference on Machine Learning , 2019. -
  165. [247][247] S. Reich, D. Mueller, and N. Andrews, 'Ensemble distillation for structured prediction: Calibrated, accurate, fast-choose three,' in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020, pp. 5583-5595. -
  166. [248][248] G. Wang, W. Li, S. Ourselin, and T. Vercauteren, 'Automatic brain tumor segmentation using convolutional neural networks with testtime augmentation,' in International MICCAI Brainlesion Workshop . Springer, 2018, pp. 61-72. -
  167. [249][249] G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren, 'Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks,' Neurocomputing , vol. 338, pp. 34-45, 2019. -
  168. [250][250] N. Moshkov, B. Mathe, A. Kertesz-Farkas, R. Hollandi, and P. Horvath, 'Test-time augmentation for deep learning-based cell segmentation on microscopy images,' Scientific reports , vol. 10, no. 1, pp. 1-7, 2020. -
  169. [251][251] O. Ronneberger, P. Fischer, and T. Brox, 'U-net: Convolutional networks for biomedical image segmentation,' in International Conference on Medical image computing and computer-assisted intervention . Springer, 2015, pp. 234-241. -
  170. [253][253] I. Kim, Y. Kim, and S. Kim, 'Learning loss for test-time augmentation,' in Advances in Neural Information Processing Systems , 2020, pp. 4163-4174. -
  171. [254][254] Q. Yu and K. Aizawa, 'Unsupervised out-of-distribution detection by maximum classifier discrepancy,' in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 9518-9526. -
  172. [255][255] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan, 'Likelihood ratios for out-of-distribution detection,' in Advances in Neural Information Processing Systems , 2019, pp. 14 707-14 718. -
  173. [258][258] J. Davis and M. Goadrich, 'The relationship between precision-recall and roc curves,' in Proceedings of the 23rd international conference on Machine learning , 2006, pp. 233-240. -
  174. [259][259] T. Pearce, A. Brintrup, M. Zaki, and A. Neely, 'High-quality prediction intervals for deep learning: A distribution-free, ensembled approach,' in International Conference on Machine Learning . PMLR, 2018, pp. 4075-4084. -
  175. [261][261] P. McClure, N. Rho, J. A. Lee, J. R. Kaczmarzyk, C. Y. Zheng, S. S. Ghosh, D. M. Nielson, A. G. Thomas, P. Bandettini, and F. Pereira, 'Knowing what you know in brain segmentation using bayesian deep neural networks,' Frontiers in neuroinformatics , vol. 13, p. 67, 2019. -
  176. [263][263] R. D. Soberanis-Mukul, N. Navab, and S. Albarqouni, 'Uncertaintybased graph convolutional networks for organ segmentation refinement,' in Medical Imaging with Deep Learning . PMLR, 2020, pp. 755-769. -
  177. [264][264] P. Seebock, J. I. Orlando, T. Schlegl, S. M. Waldstein, H. Bogunovic, S. Klimscha, G. Langs, and U. Schmidt-Erfurth, 'Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal oct,' IEEE Transactions on Medical Imaging , vol. 39, p. 87-98, 2020. -
  178. [265][265] V. Kuleshov, N. Fenner, and S. Ermon, 'Accurate uncertainties for deep learning using calibrated regression,' in International Conference on Machine Learning . PMLR, 2018, pp. 2796-2804. -
  179. [266][266] S. Seo, P. H. Seo, and B. Han, 'Learning for single-shot confidence calibration in deep neural networks through stochastic inferences,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 9030-9038. -
  180. [267][267] Z. Li and D. Hoiem, 'Improving confidence estimates for unfamiliar examples,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2020, pp. 2686-2695. -
  181. [268][268] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, 'Rethinking the inception architecture for computer vision,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2818-2826. -
  182. [270][270] R. M¨ uller, S. Kornblith, and G. E. Hinton, 'When does label smoothing help?' in Advances in Neural Information Processing Systems , 2019, pp. 4694-4703. -
  183. [272][272] P. Izmailov, W. J. Maddox, P. Kirichenko, T. Garipov, D. Vetrov, and A. G. Wilson, 'Subspace inference for bayesian deep learning,' in Uncertainty in Artificial Intelligence . PMLR, 2020, pp. 1169-1179. -
  184. [275][275] A. Mehrtash, W. M. Wells, C. M. Tempany, P. Abolmaesumi, and T. Kapur, 'Confidence calibration and predictive uncertainty estimation for deep medical image segmentation,' IEEE Transactions on Medical Imaging , 2020. -
  185. [276][276] B. Zadrozny and C. Elkan, 'Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers,' in International Conference on Machine Learning , vol. 1. Citeseer, 2001, pp. 609616. -
  186. [277][277] D. Hendrycks, M. Mazeika, and T. Dietterich, 'Deep anomaly detection with outlier exposure,' in International Conference on Learning Representations , 2019. -
  187. [278][278] S. Thulasidasan, G. Chennupati, J. A. Bilmes, T. Bhattacharya, and S. Michalak, 'On mixup training: Improved calibration and predictive uncertainty for deep neural networks,' in Advances in Neural Information Processing Systems , 2019, pp. 13 888-13 899. -
  188. [280][280] K. Patel, W. Beluch, D. Zhang, M. Pfeiffer, and B. Yang, 'On-manifold adversarial data augmentation improves uncertainty calibration,' in 2020 25th International Conference on Pattern Recognition (ICPR) . IEEE, 2021, pp. 8029-8036. -
  189. [281][281] D. Comaniciu, V. Ramesh, and P. Meer, 'Real-time tracking of nonrigid objects using mean shift,' in Proceedings IEEE Conference on Computer Vision and Pattern Recognition , vol. 2. IEEE, 2000, pp. 142-149. -
  190. [282][282] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, 'mixup: Beyond empirical risk minimization,' in International Conference on Learning Representations , 2018. -
  191. [283][283] M. P. Naeini, G. F. Cooper, and M. Hauskrecht, 'Obtaining well calibrated probabilities using bayesian binning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 2015, 2015, p. 2901. -
  192. [284][284] M. Kull, M. Perell´ o-Nieto, M. K¨ angsepp, T. de Menezes e Silva Filho, H. Song, and P. A. Flach, 'Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration,' in Advances in Neural Information Processing Systems , 2019, pp. 12 29512 305. -
  193. [286][286] J. Vaicenavicius, D. Widmann, C. Andersson, F. Lindsten, J. Roll, and T. Sch¨ on, 'Evaluating model calibration in classification,' in The 22nd International Conference on Artificial Intelligence and Statistics . PMLR, 2019, pp. 3459-3467. -
  194. [287][287] M. H. DeGroot and S. E. Fienberg, 'The comparison and evaluation of forecasters,' Journal of the Royal Statistical Society: Series D (The Statistician) , vol. 32, no. 1-2, pp. 12-22, 1983. -
  195. [288][288] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, 'Measuring calibration in deep learning,' in CVPR Workshops , 2019, pp. 38-41. -
  196. [289][289] A. Ghandeharioun, B. Eoff, B. Jou, and R. Picard, 'Characterizing sources of uncertainty to proxy calibration and disambiguate annotator and data bias,' in 2019 IEEE/CVF International Conference on Computer Vision Workshop . IEEE, 2019, pp. 4202-4206. -
  197. [290][290] F. J. Pulgar, A. J. Rivera, F. Charte, and M. J. del Jesus, 'On the impact of imbalanced data in convolutional neural networks performance,' in International Conference on Hybrid Artificial Intelligence Systems . Springer, 2017, pp. 220-232. -
  198. [291][291] K. Lee, K. Lee, H. Lee, and J. Shin, 'A simple unified framework for detecting out-of-distribution samples and adversarial attacks,' in Advances in Neural Information Processing Systems , 2018, pp. 71677177. -
  199. [295][295] M. Ghavamzadeh, S. Mannor, J. Pineau, and A. Tamar, 'Bayesian reinforcement learning: A survey,' Foundations and Trends® in Machine Learning , vol. 8, no. 5-6, pp. 359-483, 2015. -
  200. [296][296] S. Hu, D. Worrall, S. Knegt, B. Veeling, H. Huisman, and M. Welling, 'Supervised uncertainty quantification for segmentation with multiple annotations,' in International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2019, pp. 137-145. -

200 of 240 shown

Cited by

Sorted by citations

Items 201–300 of 371

Related

See all

Metadata

Authors
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., Shahzad, M., Yang, W., Bamler, R., & Zhu, X. X.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2107.03342
arXiv
2107.03342
OpenAlex
W4287083725
Cited by
232
References
0
Citations by year
2026: 2 · 2025: 10 · 2024: 36 · 2023: 113 · 2022: 65 · 2021: 6
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Bundesministerium für Bildung und Forschung (01DD20001)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
Keywords
uncertainty estimation, deep neural network, model uncertainty, data uncertainty, Bayesian neural networks, neural network ensembles, neural network calibration, safety-critical applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review