CoChat
Sign Up
Cited by 243Machine Learning in Materials ScienceComputational Drug Discovery MethodsAdversarial Robustness in Machine Learning

The DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index.

A quantitative uncertainty metric controls error in neural network-driven chemical discovery

Open access
Janet, Jon Paul, Duan, Chenru, Yang, Tzuhsiung, Nandy, Aditya, Kulik, Heather J.·2019·Chemical Science
DOI 10.1039/c9sc02298hPMC PMC6764470

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

A predictive approach for driving down machine learning model errors is introduced and demonstrated across discovery for inorganic and organic chemistry.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1039/c9sc02298h

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Schematic of an ANN annotated with the four uncertainty metrics considered in this work. Two points are compared in terms of their feature space distance (i.e., the difference between two points in the molecular representation) on a t-distributed stochastic neighbor embedding map49 (t-SNE) of data in the input layer (top, left, annotations in orange) and the latent space distance (i.e., the difference between two points in the final layer latent space) on a t-SNE of the data in the last layer (top, right, annotations in green). The standard ANN architecture (middle) is compared at bottom for Monte-Carlo dropout (i.e., zeroed out nodes) and ensemble models (i.e., varied model weights) at bottom left and right.
Fig. 1 Schematic of an ANN annotated with the four uncertainty metrics considered in this work. Two points are compared in terms of their feature space distance (i.e., the difference between two points in the molecular representation) on a t-distributed stochastic neighbor embedding map49 (t-SNE) of data in the input layer (top, left, annotations in orange) and the latent space distance (i.e., the difference between two points in the final layer latent space) on a t-SNE of the data in the last layer (top, right, annotations in green). The standard ANN architecture (middle) is compared at bottom for Monte-Carlo dropout (i.e., zeroed out nodes) and ensemble models (i.e., varied model weights) at bottom left and right.CC BY-NC · PubMed Central
Fig. 2 (left) Comparison of inorganic training and CSD test data in the dominant two principal components of the RAC-155 representation of the training data set. The density of training data is shown as gray squares shaded as indicated in inset count colorbar. CSD test data points are shown as circles colored by the 10-nearest-neighbor-averaged Euclidean distance in RAC-155 space, as shown in dist. inset color bar. Four representative high-distance structures are shown in circle insets in ball and stick representations: (top left inset, CSD ID: FATJIT) a Co(ii) complex with S/N macrocycle and axial Br– ligands, (top right inset, CSD ID: EYUSUO) Mn(ii) tetra-chlorophenyl-porphyrin with acrylamide axial ligands, (bottom left inset, CSD ID: CEDTAJ) a Mn(ii) complex with thioarsenite ligands, and (bottom right inset, CSD ID: ECODIM) an Fe(ii) complex with boronated dipyrazole and thiolated phenanthrene ligands. (right) Distribution of absolute CSD test set model errors for ΔEH-L (in kcal mol–1, bins: 2.5 kcal mol–1) with the MAE annotated as a green vertical bar and the cumulative count shown in blue according to the axis on the right.
Fig. 2 (left) Comparison of inorganic training and CSD test data in the dominant two principal components of the RAC-155 representation of the training data set. The density of training data is shown as gray squares shaded as indicated in inset count colorbar. CSD test data points are shown as circles colored by the 10-nearest-neighbor-averaged Euclidean distance in RAC-155 space, as shown in dist. inset color bar. Four representative high-distance structures are shown in circle insets in ball and stick representations: (top left inset, CSD ID: FATJIT) a Co(ii) complex with S/N macrocycle and axial Br– ligands, (top right inset, CSD ID: EYUSUO) Mn(ii) tetra-chlorophenyl-porphyrin with acrylamide axial ligands, (bottom left inset, CSD ID: CEDTAJ) a Mn(ii) complex with thioarsenite ligands, and (bottom right inset, CSD ID: ECODIM) an Fe(ii) complex with boronated dipyrazole and thiolated phenanthrene ligands. (right) Distribution of absolute CSD test set model errors for ΔEH-L (in kcal mol–1, bins: 2.5 kcal mol–1) with the MAE annotated as a green vertical bar and the cumulative count shown in blue according to the axis on the right.CC BY-NC · PubMed Central
Fig. 3 Relationship between spin-splitting ANN model errors (in kcal mol–1) on a 116 molecule CSD set and three uncertainty metrics all in kcal mol–1: latent model energetic, calibrated std. dev. (left), mc-dropout std. dev. (middle), and 10-model ensemble std. dev. (right). The translucent green region corresponds to one std. dev. and translucent yellow to two std. dev. The points with model errors that lie inside either of these two bounds are shown in black, and the percentage within the green or yellow regions are annotated in each graph in green and yellow, respectively. The points outside two std. dev. are colored red, and the percentage of points in this group is annotated in each graph in red. Three points are omitted from the ensemble plot to allow for a consistent x-axis range.
Fig. 3 Relationship between spin-splitting ANN model errors (in kcal mol–1) on a 116 molecule CSD set and three uncertainty metrics all in kcal mol–1: latent model energetic, calibrated std. dev. (left), mc-dropout std. dev. (middle), and 10-model ensemble std. dev. (right). The translucent green region corresponds to one std. dev. and translucent yellow to two std. dev. The points with model errors that lie inside either of these two bounds are shown in black, and the percentage within the green or yellow regions are annotated in each graph in green and yellow, respectively. The points outside two std. dev. are colored red, and the percentage of points in this group is annotated in each graph in red. Three points are omitted from the ensemble plot to allow for a consistent x-axis range.CC BY-NC · PubMed Central
Fig. 4 Model errors (in kcal mol–1) for 127k QM9 atomization energy test points shown as contours as a function of uncertainty metrics. The three uncertainty metrics compared are: feature space distance (in arb. u., left, with top left color bar), latent space distance (in arb. u., middle, with top right color bar), and 10-model ensemble std. dev. (in kcal mol–1, with top right color bar). One standard deviation cutoffs are shown as orange lines for the latent space distance from the calibrated error model (center) and directly from the ensemble (right).
Fig. 4 Model errors (in kcal mol–1) for 127k QM9 atomization energy test points shown as contours as a function of uncertainty metrics. The three uncertainty metrics compared are: feature space distance (in arb. u., left, with top left color bar), latent space distance (in arb. u., middle, with top right color bar), and 10-model ensemble std. dev. (in kcal mol–1, with top right color bar). One standard deviation cutoffs are shown as orange lines for the latent space distance from the calibrated error model (center) and directly from the ensemble (right).CC BY-NC · PubMed Central
Fig. 5 MAE for predicted points (inside cutoff, green squares) and those not predicted (outside cutoff, orange squares) compared to the training data MAE (blue horizontal dashed line) along with data fraction in each set for the inorganic CSD test set (left) and organic QM9 set (right). The most distant point in the test set is scaled to have a latent distance of 1.0 for comparison across data sets but the x-axis range is then truncated to focus on the range of latent distance cutoffs that affect most of the data.
Fig. 5 MAE for predicted points (inside cutoff, green squares) and those not predicted (outside cutoff, orange squares) compared to the training data MAE (blue horizontal dashed line) along with data fraction in each set for the inorganic CSD test set (left) and organic QM9 set (right). The most distant point in the test set is scaled to have a latent distance of 1.0 for comparison across data sets but the x-axis range is then truncated to focus on the range of latent distance cutoffs that affect most of the data.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 9

  1. van der Maaten, 2008, J. Mach. Learn. Res., 9, 2579
  2. Y. Gal and Z.Ghahramani , in Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, international conference on machine learning , 2016 , pp. 1050–1059
  3. Duvenaud, 2015, Adv. Neural Inf. Process. Syst., 2215
  4. Larsen, 2017, J. Phys.: Condens. Matter, 29, 273002
  5. J. H. Metzen , T.Genewein , V.Fischer and B.Bischoff , On Detecting Adversarial Perturbations , in 5th International Conference on Learning Representations , ICLR , 2017
  6. T. Schlegl , P.Seeböck , S. M.Waldstein , U.Schmidt-Erfurth and G.Langs , in Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery, Information Processing in Medical Imaging , Springer International Publishing , 2017 , pp. 146–157
  7. Breiman, 1996, Mach. Learn., 24, 123
  8. C. C. Aggarwal , A.Hinneburg and D. A.Keim , in On the Surprising Behavior of Distance Metrics in High Dimensional Space, Database Theory—ICDT 2001 , ed. J. Van den Bussche and V. Vianu , Springer Berlin Heidelberg , Berlin, Heidelberg , 2001 , pp. 420–434
  9. Broto, 1984, Eur. J. Med. Chem., 19, 71

Cited by

Sorted by citations

See all 243

Related

See all

Metadata

Authors
Janet, J. P., Duan, C., Yang, T., Nandy, A., & Kulik, H. J.
Affiliations
Massachusetts Institute of Technology (US)
Corresponding author
Kulik, Heather J.
ORCID
Venue
Chemical Science
Publisher
Royal Society of Chemistry (RSC)
Published
2019
Volume
10
Issue
34
Pages
7913-7922
ISSN
  • 2041-6520 (ISSN-L)
  • 2041-6539
DOI
10.1039/c9sc02298h
PMC
PMC6764470
OpenAlex
W2959623824
Cited by
243
References
87
Citations by year
2026: 43 · 2025: 29 · 2024: 34 · 2023: 23 · 2022: 37 · 2021: 39 · 2020: 29 · 2019: 8
FWCI
10.20
Type
journal article
Language
en
Open access
Licence
Locations
  • Chemical Science (publishedVersion, cc-by-nc)
  • ChemRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • Figshare (submittedVersion, cc-by-nc-nd)
Funders
  • Office of Naval Research Global (N00014-18-1-2434)
  • Burroughs Wellcome Fund (Career Award at the Scientific Interface)
  • American Association for the Advancement of Science (Marion Milligan Mason Award)
  • Defense Advanced Research Projects Agency (D18AP00039)
Topics
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
uncertainty quantification, neural network model, chemical discoveries, prediction error, active learning (machine learning), chemical space exploration, model calibration, inorganic chemistry, organic chemistry

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review