The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Uncertainty Quantification Using Neural Networks for Molecular Property Prediction
Open accessAbstract
Uncertainty quantification (UQ) is an important component of molecular property prediction, particularly for drug discovery applications where model predictions direct experimental design and where unanticipated imprecision wastes valuable time and resources. The need for UQ is especially acute for neural models, which are becoming increasingly standard yet are challenging to interpret. While several approaches to UQ have been proposed in the literature, there is no clear consensus on the comparative performance of these models. In this paper, we study this question in the context of regression tasks. We systematically evaluate several methods on five regression data sets using multiple complementary performance metrics. Our experiments show that none of the methods we tested is unequivocally superior to all others, and none produces a particularly reliable ranking of errors across multiple data sets. While we believe that these results show that existing UQ methods are not sufficient for all common use cases and further research is needed, we conclude with a practical recommendation as to which existing techniques seem to perform well relative to others.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Neural Message Passing for Quantum Chemistry
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01212 · Cited by 3,440
MoleculeNet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018) · Chemical Science · doi:10.1039/c7sc02664a · Cited by 3,107
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Popular Ensemble Methods: An Empirical Study
Opitz, D., & Maclin, R. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.614 · Cited by 2,953
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_16 · Cited by 2,915
QSAR Modeling: Where have you been? Where are you going to?
Cherkasov, A., Muratov, E. N., Fourches, D., Varnek, A., Baskin, I. I., Cronin, M., Dearden, J., Gramatica, P., Martin, Y. C., Todeschini, R., et al. (2014) · Journal of Medicinal Chemistry · doi:10.1021/jm4004285 · Cited by 2,164
Analyzing Learned Molecular Representations for Property Prediction
Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00237 · Cited by 1,923
Self-citationBreiman, L. (1996) · Machine Learning · doi:10.1007/bf00117832 · Cited by 1,912
Molecular graph convolutions: moving beyond fingerprints
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., & Riley, P. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9938-8 · Cited by 1,631
Prediction of Physicochemical Parameters by Atomic Contributions
Wildman, S. A., & Crippen, G. M. (1999) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci990307l · Cited by 1,464
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
Muratov, E. N., Bajorath, J., Sheridan, R. P., Tetko, I. V., Filimonov, D., Poroikov, V., Oprea, T. I., Baskin, I. I., Varnek, A., Roitberg, A., et al. (2020) · Chemical Society Reviews · doi:10.1039/d0cs00098a · Cited by 913
On a simple approach for determining applicability domain of QSAR models
Roy, K., Kar, S., & Ambure, P. (2015) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2015.04.013 · Cited by 801
Estimating the mean and variance of the target probability distribution
Nix, D. A., & Weigend, A. S. (1994) · Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) · doi:10.1109/icnn.1994.374138 · Cited by 700
Convolutional embedding of attributed molecular graphs for physical property prediction
Coley, C. W., Barzilay, R., Green, W. H., Jaakkola, T. S., & Jensen, K. F. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00601 · Cited by 470
Self-citationExploring Activity Cliffs in Medicinal Chemistry: Miniperspective
Stumpfe, D., & Bajorath, J. (2012) · Journal of Medicinal Chemistry · doi:10.1021/jm201706b · Cited by 374
Similarity to Molecules in the Training Set Is a Good Discriminator for Prediction Accuracy in QSAR
Sheridan, R. P., Feuston, B. P., Maiorov, V. N., & Kearsley, S. K. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci049782w · Cited by 298
OpitzDavid, & MaclinRichard (1999) · doi:10.5555/3013545.3013549 · Cited by 293
A quantitative uncertainty metric controls error in neural network-driven chemical discovery
Janet, J. P., Duan, C., Yang, T., Nandy, A., & Kulik, H. J. (2019) · Chemical Science · doi:10.1039/c9sc02298h · Cited by 243
(1997) · Springer Series in Statistics · doi:10.1007/978-1-4612-0667-5 · Cited by 211
Methods for comparing uncertainty quantifications for material property predictions
Tran, K., Neiswanger, W., Yoon, J., Zhang, Q., Xing, E., & Ulissi, Z. W. (2020) · Machine Learning: Science and Technology · doi:10.1088/2632-2153/ab7e1a · Cited by 205
Zhang, Y., & Lee, A. A. (2019) · Chemical Science · doi:10.1039/c9sc00616h · Cited by 162
Aniceto, N., Freitas, A. A., Bender, A., & Ghafourian, T. (2016) · Journal of Cheminformatics · doi:10.1186/s13321-016-0182-y · Cited by 151
Cortés‐Ciriano, I., & Bender, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00542 · Cited by 105
Liu, R., & Wallqvist, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00597 · Cited by 78
Schroeter, T. S., Schwaighofer, A., Mika, S., ter Laak, A., Suelzle, D., Ganzer, U., Heinrich, N., & Müller, K. (2007) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9125-z · Cited by 70
Toplak, M., Močnik, R., Polajnar, M., Bosnić, Z., Carlsson, L., Hasselgren, C., Demšar, J., Boyer, S. K., Zupan, B., & Stålring, J. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci4006595 · Cited by 67
Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
Cortés-Ciriano, I., & Bender, A. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00297 · Cited by 51
Scalable Gaussian process regression using deep neural networks
Huang, W., Zhao, D., Sun, F., Liu, H., & Chang, E. Y. (2015) · Cited by 50
Graph Neural Processes: Towards Bayesian Graph Neural NetworksPreprint
Carr, A. N., & Wingate, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10042 · Cited by 17
A comprehensive study on the prediction reliability of graph neural networks for virtual screeningPreprint
Yang, S., Lee, K.-H., & Ryu, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07611 · Cited by 8
Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property PredictionPreprint
Scalia, G., Grambow, C. A., Pernici, B., Li, Y., & Green, W. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03127 · Cited by 6
Cited by
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063
Machine Learning Methods for Small Data Challenges in Molecular Science
Dou, B., Zhu, Z., Merkurjev, E., Ke, L., Chen, L., Jiang, J., Zhu, Y., Liu, J., Zhang, B., & Wei, G.-W. (2023) · Chemical Reviews · doi:10.1021/acs.chemrev.3c00189 · Cited by 499
Corso, G., Stark, H., Jegelka, S., Jaakkola, T., & Barzilay, R. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00294-7 · Cited by 454
Self-citationApplications of Deep Learning in Molecule Generation and Molecular Property Prediction
Walters, W. P., & Barzilay, R. (2021) · Accounts of Chemical Research · doi:10.1021/acs.accounts.0c00699 · Cited by 382
Self-citationGenerative models for molecular discovery: Recent advances and challenges
Bilodeau, C., Jin, W., Jaakkola, T., Barzilay, R., & Jensen, K. F. (2022) · WIREs Computational Molecular Science · doi:10.1002/wcms.1608 · Cited by 318
Self-citation
Metadata
- Authors
- Hirschfeld, L., Swanson, K., Yang, K., Barzilay, R., & Coley, C. W.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2020-07-23
- Volume
- 60
- Issue
- 8
- Pages
- 3770-3780
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W3028499909
- Cited by
- 334
- References
- 36
- FWCI
- 16.56
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.