The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction
Open accessAbstract
Advances in deep neural network (DNN)-based molecular property prediction have recently led to the development of models of remarkable accuracy and generalization ability, with graph convolutional neural networks (GCNNs) reporting state-of-the-art performance for this task. However, some challenges remain, and one of the most important that needs to be fully addressed concerns uncertainty quantification. DNN performance is affected by the volume and the quality of the training samples. Therefore, establishing when and to what extent a prediction can be considered reliable is just as important as outputting accurate predictions, especially when out-of-domain molecules are targeted. Recently, several methods to account for uncertainty in DNNs have been proposed, most of which are based on approximate Bayesian inference. Among these, only a few scale to the large data sets required in applications. Evaluating and comparing these methods has recently attracted great interest, but results are generally fragmented and absent for molecular property prediction. In this paper, we quantitatively compare scalable techniques for uncertainty estimation in GCNNs. We introduce a set of quantitative criteria to capture different uncertainty aspects and then use these criteria to compare MC-dropout, Deep Ensembles, and bootstrapping, both theoretically in a unified framework that separates aleatoric/epistemic uncertainty and experimentally on public data sets. Our experiments quantify the performance of the different uncertainty estimation methods and their impact on uncertainty-related error reduction. Our findings indicate that Deep Ensembles and bootstrapping consistently outperform MC-dropout, with different context-specific pros and cons. Our analysis leads to a better understanding of the role of aleatoric/epistemic uncertainty, also in relation to the target data set features, and highlights the challenge posed by out-of-domain uncertainty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Cited by 33,419
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Cipolla, R., Gal, Y., & Kendall, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00781 · Cited by 3,151
MoleculeNet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018) · Chemical Science · doi:10.1039/c7sc02664a · Cited by 3,107
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639
Convolutional Networks on Graphs for Learning Molecular Fingerprints
Duvenaud, D., Maclaurin, D., Aguilera-Iparraguirre, J., Gómez-Bombarelli, R., Hirzel, T., Aspuru-Guzik, A., & Adams, R. P. (2015) · arXiv · doi:10.48550/arxiv.1509.09292 · Cited by 2,628
Quantum chemistry structures and properties of 134 kilo molecules
Ramakrishnan, R., Dral, P. O., Rupp, M., & von Lilienfeld, O. A. (2014) · Scientific Data · doi:10.1038/sdata.2014.22 · Cited by 1,966
Analyzing Learned Molecular Representations for Property Prediction
Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00237 · Cited by 1,923
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Probabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00587.x · Cited by 1,783
(1994) · Aston Publications Explorer (Aston University) · Cited by 1,245
A clinically applicable approach to continuous prediction of future acute kidney injury
Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., et al. (2019) · Nature · doi:10.1038/s41586-019-1390-1 · Cited by 1,086
Practical Variational Inference for Neural Networks
Graves, A. (2011) · Cited by 1,078
Obtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Less is more: Sampling chemical space with active learning
Smith, J. S., Nebgen, B., Lubbers, N., Isayev, O., & Roitberg, A. E. (2018) · The Journal of Chemical Physics · doi:10.1063/1.5023802 · Cited by 811
Estimating the mean and variance of the target probability distribution
Nix, D. A., & Weigend, A. S. (1994) · Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) · doi:10.1109/icnn.1994.374138 · Cited by 700
Smith, J. S., Nebgen, B. T., Zubatyuk, R., Lubbers, N., Devereux, C., Barros, K., Tretiak, S., Isayev, O., & Roitberg, A. E. (2019) · Nature Communications · doi:10.1038/s41467-019-10827-4 · Cited by 664
The Power of Ensembles for Active Learning in Image Classification
Beluch, W. H., Genewein, T., Nurnberger, A., & Kohler, J. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00976 · Cited by 655
The Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604
Large-scale comparison of machine learning methods for drug target prediction on ChEMBL
Mayr, A., Klambauer, G., Unterthiner, T., Steijaert, M., Wegner, J. K., Ceulemans, H., Clevert, D.-A., & Hochreiter, S. (2018) · Chemical Science · doi:10.1039/c8sc00148k · Cited by 584
Convolutional embedding of attributed molecular graphs for physical property prediction
Coley, C. W., Barzilay, R., Green, W. H., Jaakkola, T. S., & Jensen, K. F. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00601 · Cited by 470
Self-citationAccurate Uncertainties for Deep Learning Using Calibrated Regression
Kuleshov, V., Fenner, N., & Ermon, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00263 · Cited by 337
Evaluating Bayesian Deep Learning Methods for Semantic SegmentationPreprint
Mukhoti, J., & Gal, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12709 · Cited by 272
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision
Gustafsson, F. K., Danelljan, M., & Schon, T. B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00167 · Cited by 257
Uncertainty Estimates and Multi-hypotheses Networks for Optical Flow
Ilg, E., Çiçek, Ö., Galesso, S., Klein, A., Makansi, O., Hutter, F., & Brox, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_40 · Cited by 213
A Complete Recipe for Stochastic Gradient MCMC
Ma, Y.-A., Chen, T., & Fox, E. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04696 · Cited by 175
Heteroscedastic Gaussian process regression
Le, Q. V., Smola, A. J., & Canu, S. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102413 · Cited by 175
Addressing uncertainty in atomistic machine learning
Peterson, A. A., Christensen, R., & Khorshidi, A. (2017) · Physical Chemistry Chemical Physics · doi:10.1039/c7cp00375g · Cited by 173
Evaluating and Calibrating Uncertainty Prediction in Regression Tasks
Levi, D., Gispan, L., Giladi, N., & Fetaya, E. (2022) · Sensors · doi:10.3390/s22155540 · Cited by 169
Zhang, Y., & Lee, A. A. (2019) · Chemical Science · doi:10.1039/c9sc00616h · Cited by 162
Ryu, S., Kwon, Y., & Kim, W. Y. (2019) · Chemical Science · doi:10.1039/c9sc01992h · Cited by 153
Concrete DropoutPreprint
Gal, Y., Hron, J., & Kendall, A. (2017) · arXiv (Cornell University) · Cited by 129
Choi, S., Lee, K., Lim, S., & Oh, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462978 · Cited by 119
Alchemy: A Quantum Chemistry Dataset for Benchmarking AI ModelsPreprint
Chen, G., Chen, P., Hsieh, C., Lee, C., Liao, B., Liao, R., Liu, W., Qiu, J., Sun, Q., Tang, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09427 · Cited by 116
Grambow, C. A., Li, Y., & Green, W. H. (2019) · The Journal of Physical Chemistry A · doi:10.1021/acs.jpca.9b04195 · Cited by 109
Self-citationCortés‐Ciriano, I., & Bender, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00542 · Cited by 105
Early Stopping as Nonparametric Variational Inference
Duvenaud, D., Maclaurin, D., & Adams, R. P. (2016) · International Conference on Artificial Intelligence and Statistics · Cited by 60
Reliable Estimation of Prediction Uncertainty for Physicochemical Property Models
Proppe, J., & Reiher, M. (2017) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.7b00235 · Cited by 58
Self-Evolving Machine: A Continuously Improving Model for Molecular Thermochemistry
Li, Y., Han, K., Grambow, C. A., & Green, W. H. (2019) · The Journal of Physical Chemistry A · doi:10.1021/acs.jpca.8b10789 · Cited by 56
Self-citationLiu, R., Glover, K., Feasel, M. G., & Wallqvist, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00114 · Cited by 44
Uncertainty Estimates and Multi-Hypotheses Networks for Optical FlowPreprint
Ilg, E., Çiçek, Ö., Galesso, S., Klein, A., Makansi, O., Hutter, F., & Brox, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07095 · Cited by 28
Predictive Uncertainty Quantification with Compound Density NetworksPreprint
Kristiadi, A., Däubener, S., & Fischer, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01080 · Cited by 18
Pearce, T., Anastassacos, N., Zaki, M., & Neely, A. D. (2018) · arXiv (Cornell University) · Cited by 15
Cited by
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,406
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063
Chemprop: A Machine Learning Package for Chemical Property Prediction
Heid, E., Greenman, K. P., Chung, Y., Li, S.-C., Graff, D. E., Vermeire, F. H., Wu, H., Green, W. H., & McGill, C. J. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01250 · Cited by 663
Self-citationMachine Learning in Chemical Engineering: Strengths, Weaknesses, Opportunities, and Threats
Dobbelaere, M. R., Plehiers, P. P., Van de Vijver, R., Stevens, C. V., & Van Geem, K. M. (2021) · Engineering · doi:10.1016/j.eng.2021.03.019 · Cited by 323
Generative models for molecular discovery: Recent advances and challenges
Bilodeau, C., Jin, W., Jaakkola, T., Barzilay, R., & Jensen, K. F. (2022) · WIREs Computational Molecular Science · doi:10.1002/wcms.1608 · Cited by 318
Metadata
- Authors
- Scalia, G., Grambow, C. A., Pernici, B., Li, Y.-P., & Green, W. H.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2020-04-03
- Volume
- 60
- Issue
- 6
- Pages
- 2697-2717
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W3014339631
- Cited by
- 269
- References
- 49
- FWCI
- 13.94
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.