Cited by 47Computational Drug Discovery MethodsMachine Learning in Materials ScienceAnimal testing and alternatives

Assessing Deep and Shallow Learning Methods for Quantitative Prediction of Acute Chemical Toxicity

Open access
Liu, Ruifeng, Madore, Michael, Glover, Kyle P, Feasel, Michael G, Wallqvist, Anders·2018·Toxicological Sciences

Abstract

Animal-based methods for assessing chemical toxicity are struggling to meet testing demands. In silico approaches, including machine-learning methods, are promising alternatives. Recently, deep neural networks (DNNs) were evaluated and reported to outperform other machine-learning methods for quantitative structure-activity relationship modeling of molecular properties. However, most of the reported performance evaluations relied on global performance metrics, such as the root mean squared error (RMSE) between the predicted and experimental values of all samples, without considering the impact of sample distribution across the activity spectrum. Here, we carried out an in-depth analysis of DNN performance for quantitative prediction of acute chemical toxicity using several datasets. We found that the overall performance of DNN models on datasets of up to 30 000 compounds was similar to that of random forest (RF) models, as measured by the RMSE and correlation coefficients between the predicted and experimental results. However, our detailed analyses demonstrated that global performance metrics are inappropriate for datasets with a highly uneven sample distribution, because they show a strong bias for the most populous compounds along the toxicity spectrum. For highly toxic compounds, DNN and RF models trained on all samples performed much worse than the global performance metrics indicated. Surprisingly, our variable nearest neighbor method, which utilizes only structurally similar compounds to make predictions, performed reasonably well, suggesting that information of close near neighbors in the training sets is a key determinant of acute toxicity predictions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/toxsci/kfy111

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Liu et al. found that RF and DNN performed better on the global dataset, while k-NN performed better on the unbalanced acute toxicity datasets, using ECFP uniformly.

    “It was found that RF and DNN performed better on the global dataset, while k-NN performed better on the unbalanced acute toxicity datasets. This result also highlights the importance of neighbor information in acute toxicity prediction [126].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Liu, R., Madore, M., Glover, K. P., Feasel, M. G., & Wallqvist, A.
Affiliations
  • Telemedicine & Advanced Technology Research Center (US)
  • Defense Threat Reduction Agency (US)
  • U.S. Army Combat Capabilities Development Command Chemical Biological Center (US)
Corresponding author
  • Liu, Ruifeng
  • Wallqvist, Anders
Venue
Toxicological Sciences
Published
2018-05-02
Volume
164
Issue
2
Pages
512-526
ISSN
  • 1096-0929 (ISSN-L)
  • 1096-6080
OpenAlex
W2802630768
Cited by
47
References
20
Citations by year
2026: 1 · 2025: 4 · 2024: 5 · 2023: 10 · 2022: 4 · 2021: 6 · 2020: 8 · 2019: 6 · 2018: 3
FWCI
4.29
Type
journal article
Language
en
Open access
Locations
  • Toxicological Sciences (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • U.S. Army Medical Research
  • Defense Threat Reduction Agency (CBCall14-CBS-05-2-0007)
Topics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
Keywords
deep neural network, random forest, QSAR, in silico toxicity prediction, alternatives to animal testing
MeSH terms
Machine Learning, Deep Learning, Animals, Rabbits, Neural Networks, Computer, Toxicity Tests, Computational Biology, Quantitative Structure-Activity Relationship, Mice, Rats, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review