CoChat
Sign Up
PreprintCited by 8Adversarial Robustness in Machine LearningMachine Learning and Data ClassificationGaussian Processes and Bayesian Inference

The DOI matches its DataCite record.

Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles

Open access
Jain, Siddhartha, Liu, Ge, Mueller, Jonas, Gifford, David·2019·arXiv
DOI 10.48550/arxiv.1906.07380arXiv 1906.07380
Open this paper in CoChat — free account

Abstract

The inaccuracy of neural network models on inputs that do not stem from the training data distribution is both problematic and at times unrecognized. Model uncertainty estimation can address this issue, where uncertainty estimates are often based on the variation in predictions produced by a diverse ensemble of models applied to the same input. Here we describe Maximize Overall Diversity (MOD), a straightforward approach to improve ensemble-based uncertainty estimates by encouraging larger overall diversity in ensemble predictions across all possible inputs that might be encountered in the future. When applied to various neural network ensembles, MOD significantly improves predictive performance for out-of-distribution test examples without sacrificing in-distribution performance on 38 Protein-DNA binding regression datasets, 9 UCI datasets, and the IMDB-Wiki image dataset. Across many Bayesian optimization tasks, the performance of UCB acquisition is also greatly improved by leveraging MOD uncertainty estimates.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1906.07380

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 26

  1. [Balan et al. 2015] Balan, A. K.; Rathod, V.; Murphy, K. P.; and Welling, M. 2015. Bayesian dark knowledge. In Advances in Neural Information Processing Systems .
  2. [Barrera et al. 2016] Barrera, L. A.; Vedenko, A.; Kurland, J. V.; Rogers, J. M.; Gisselbrecht, S. S.; Rossin, E. J.; Woodard, J.; Mariani, L.; Kock, K. H.; Inukai, S.; et al. 2016. Survey of variation in human transcription factors reveals prevalent DNA binding changes. Science 351(6280):14501454.
  3. [Beluch et al. 2018] Beluch, W. H.; Genewein, T.; N¨ urnberger, A.; and K¨ ohler, J. M. 2018. The power of ensembles for active learning in image classification. In IEEE Conference on Computer Vision and Pattern Recognition .
  4. Kavukcuoglu, K.; and Wierstra, D. 2015. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424 . · arXiv
  5. [Brown 2004] Brown, G. 2004. Diversity in neural network ensembles . Ph.D. Dissertation, University of Birmingham.
  6. [Chen et al. 2017] Chen, R. Y.; Sidor, S.; Abbeel, P.; and Schulman, J. 2017. UCB exploration via Q-ensembles. arXiv:1706.01502 . · arXiv
  7. [Choi and Jang 2018] Choi, H., and Jang, E. 2018. Generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392 . · arXiv
  8. [Hafner et al. 2018] Hafner, D.; Tran, D.; Lillicrap, T.; Irpan, A.; and Davidson, J. 2018. Reliable uncertainty estimates in deep neural networks using noise contrastive priors. arXiv:1807.09289 . · arXiv
  9. [Hashimoto, Yadlowsky, and Duchi 2018] Hashimoto, T. B.; Yadlowsky, S.; and Duchi, J. C. 2018. Derivative free optimization via repeated classification. In International Conference on Artificial Intelligence and Statistics .
  10. [Hern´ andez-Lobato et al. 2017] Hern´ andez-Lobato, J. M.; Requeima, J.; Pyzer-Knapp, E. O.; and Aspuru-Guzik, A. 2017. Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. arXiv:1706.01825 . · arXiv
  11. [Lakshminarayanan, Pritzel, and Blundell 2017] Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems .
  12. [Lee et al. 2015] Lee, S.; Purushwalkam, S.; Cogswell, M.; Crandall, D.; and Batra, D. 2015. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv:1511.06314 . · arXiv
  13. [Lee et al. 2018] Lee, K.; Lee, H.; Lee, K.; and Shin, J. 2018. Training confidence-calibrated classifiers for detecting out-ofdistribution samples. In International Conference on Learning Representations .
  14. [Liang, Li, and Srikant 2017] Liang, S.; Li, Y.; and Srikant, R. 2017. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 . · arXiv
  15. [Liu and Yao 1999] Liu, Y., and Yao, X. 1999. Ensemble learning via negative correlation. Neural networks 12(10):1399-1404.
  16. [Malinin and Gales 2018] Malinin, A., and Gales, M. 2018. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems , 70477058.
  17. [Osband et al. 2016] Osband, I.; Blundell, C.; Pritzel, A.; and Van Roy, B. 2016. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems .
  18. [Papadopoulos, Edwards, and Murray 2001] Papadopoulos, G.; Edwards, P. J.; and Murray, A. F. 2001. Confidence estimation methods for neural networks: A practical comparison. IEEE Transactions on Neural Networks 12:1278-1287.
  19. [Papernot and McDaniel 2018] Papernot, N., and McDaniel, P. 2018. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765 . · arXiv
  20. [Pearce et al. 2018] Pearce, T.; Zaki, M.; Brintrup, A.; and Neel, A. 2018. Uncertainty in neural networks: Bayesian ensembling. arXiv preprint arXiv:1810.05546 . · arXiv
  21. [Ren et al. 2019] Ren, J.; Liu, P. J.; Fertig, E.; Snoek, J.; Poplin, R.; DePristo, M. A.; Dillon, J. V.; and Lakshminarayanan, B. 2019. Likelihood ratios for out-of-distribution detection. arXiv preprint arXiv:1906.02845 . · arXiv
  22. [Riquelme, Tucker, and Snoek 2018] Riquelme, C.; Tucker, G.; and Snoek, J. 2018. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations .
  23. [Rothe, Timofte, and Van Gool 2015] Rothe, R.; Timofte, R.; and Van Gool, L. 2015. Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE International Conference on Computer Vision Workshops , 10-15.
  24. [Shui et al. 2018] Shui, C.; Mozafari, A. S.; Marek, J.; Hedhli, I.; and Gagne, C. 2018. Diversity regularization in deep ensembles. arXiv:1802.07881 . · arXiv
  25. [Vyas et al. 2018] Vyas, A.; Jammalamadaka, N.; Zhu, X.; Das, D.; Kaul, B.; and Willke, T. L. 2018. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV) , 550-564.
  26. [Zagoruyko and Komodakis 2016] Zagoruyko, S., and Komodakis, N. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 . · arXiv

Cited by

Sorted by citations

See all 8

Metadata

Authors
Jain, S., Liu, G., Mueller, J., & Gifford, D.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Amazon (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
34
Issue
04
Pages
4264-4271
ISSN
2159-5399 (ISSN-L)
DOI
10.48550/arxiv.1906.07380
arXiv
1906.07380
OpenAlex
W2950003830
Cited by
8
References
0
Citations by year
2025: 2 · 2024: 1 · 2023: 2 · 2022: 1 · 2021: 2
FWCI
0.48
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
Keywords
ensemble deep learning, uncertainty estimation, ensemble diversity, out-of-distribution detection, uncertainty calibration, adversarial training, protein-DNA binding prediction, Bayesian optimization, regression analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review