Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Prediction Errors for Deep Neural Networks
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Deep learning architectures have proved versatile in a number of drug discovery applications, including the modeling of in vitro compound activity. While controlling for prediction confidence is essential to increase the trust, interpretability, and usefulness of virtual screening models in drug discovery, techniques to estimate the reliability of the predictions generated with deep learning networks remain largely underexplored. Here, we present Deep Confidence, a framework to compute valid and efficient confidence intervals for individual predictions using the deep learning technique Snapshot Ensembling and conformal prediction. Specifically, Deep Confidence generates an ensemble of deep neural networks by recording the network parameters throughout the local minima visited during the optimization phase of a single neural network. This approach serves to derive a set of base learners (i.e., snapshots) with comparable predictive power on average that will however generate slightly different predictions for a given instance. The variability across base learners and the validation residuals are in turn harnessed to compute confidence intervals using the conformal prediction framework. Using a set of 24 diverse IC 50 data sets from ChEMBL 23, we show that Snapshot Ensembles perform on par with Random Forest (RF) and ensembles of independently trained deep neural networks. In addition, we find that the confidence regions predicted using the Deep Confidence framework span a narrower set of values. Overall, Deep Confidence represents a highly versatile error prediction framework that can be applied to any deep learning-based application at no extra computational cost.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
ChEMBL: a large-scale bioactivity database for drug discovery
Gaulton, A., Bellis, L. J., Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y., McGlinchey, S., Michalovich, D., Al-Lazikani, B., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr777 · Cited by 4,382
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., & Aspuru-Guzik, A. (2018) · ACS Central Science · doi:10.1021/acscentsci.7b00572 · Cited by 3,106
QSAR Modeling: Where have you been? Where are you going to?
Cherkasov, A., Muratov, E. N., Fourches, D., Varnek, A., Baskin, I. I., Cronin, M., Dearden, J., Gramatica, P., Martin, Y. C., Todeschini, R., et al. (2014) · Journal of Medicinal Chemistry · doi:10.1021/jm4004285 · Cited by 2,164 · cited 1×
“Defining the set of molecules for which a model can generate reliable predictions has been (and still is 26 ) an area of intense research because an accurate estimation of prediction errors is essential for a model to inform decision-making in drug discovery 27–30 .”
BackgroundPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
The rise of deep learning in drug discovery
Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., & Blaschke, T. (2018) · Drug Discovery Today · doi:10.1016/j.drudis.2018.01.039 · Cited by 1,770
Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks
Segler, M. H. S., Kogej, T., Tyrchan, C., & Waller, M. P. (2018) · ACS Central Science · doi:10.1021/acscentsci.7b00512 · Cited by 1,537
Deep Neural Nets as a Method for Quantitative Structure–Activity Relationships
Ma, J., Sheridan, R. P., Liaw, A., Dahl, G. E., & Svetnik, V. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/ci500747n · Cited by 1,159 · cited 3×
“While diverse deep learning architectures have been developed to predict compound activity as point predictions 12,18,23 , modelling prediction uncertainty for small molecules using deep learning still remains an unexplored area, and this is precisely what we address in this contribution.”
BackgroundMethod
Cited by
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063
Opportunities and Challenges for Machine Learning in Materials Science
Morgan, D., & Jacobs, R. (2020) · Annual Review of Materials Research · doi:10.1146/annurev-matsci-070218-010015 · Cited by 549
Nandy, A., Duan, C., Taylor, M. G., Liu, F., Steeves, A. H., & Kulik, H. J. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00347 · Cited by 358
Constrained Bayesian optimization for automatic chemical design using variational autoencoders
Griffiths, R.-R., & Hernández-Lobato, J. M. (2020) · Chemical Science · doi:10.1039/c9sc04026a · Cited by 339
Uncertainty Quantification Using Neural Networks for Molecular Property Prediction
Hirschfeld, L., Swanson, K., Yang, K., Barzilay, R., & Coley, C. W. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00502 · Cited by 334
Scalia, G., Grambow, C. A., Pernici, B., Li, Y.-P., & Green, W. H. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00975 · Cited by 269
A quantitative uncertainty metric controls error in neural network-driven chemical discovery
Janet, J. P., Duan, C., Yang, T., Nandy, A., & Kulik, H. J. (2019) · Chemical Science · doi:10.1039/c9sc02298h · Cited by 243
Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107441 · Cited by 239
Uncertainty-aware deep learning in healthcare: A scoping review
Loftus, T. J., Shickel, B., Ruppert, M. M., Balch, J. A., Ozrazgat-Baslanti, T., Tighe, P. J., Efron, P. A., Hogan, W. R., Rashidi, P., Upchurch, G. R., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000085 · Cited by 73
“those reported in their Deep Confidence work [24].”
BackgroundReliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
Cortés-Ciriano, I., & Bender, A. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00297 · Cited by 51
“These results is line with previous results obtained for DNN-based Conformal Predictors generated using the Deep Confidence framework 31 , which is based on Snapshot Ensembles 63 .”
ResultSelf-citation
Metadata
- Authors
- Cortés-Ciriano, I., & Bender, A.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2018-10-17
- Volume
- 59
- Issue
- 3
- Pages
- 1269-1281
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2890097032
- Cited by
- 104
- References
- 19
- FWCI
- 9.59
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.