Assessing Deep and Shallow Learning Methods for Quantitative Prediction of Acute Chemical Toxicity
Open accessAbstract
Animal-based methods for assessing chemical toxicity are struggling to meet testing demands. In silico approaches, including machine-learning methods, are promising alternatives. Recently, deep neural networks (DNNs) were evaluated and reported to outperform other machine-learning methods for quantitative structure-activity relationship modeling of molecular properties. However, most of the reported performance evaluations relied on global performance metrics, such as the root mean squared error (RMSE) between the predicted and experimental values of all samples, without considering the impact of sample distribution across the activity spectrum. Here, we carried out an in-depth analysis of DNN performance for quantitative prediction of acute chemical toxicity using several datasets. We found that the overall performance of DNN models on datasets of up to 30 000 compounds was similar to that of random forest (RF) models, as measured by the RMSE and correlation coefficients between the predicted and experimental results. However, our detailed analyses demonstrated that global performance metrics are inappropriate for datasets with a highly uneven sample distribution, because they show a strong bias for the most populous compounds along the toxicity spectrum. For highly toxic compounds, DNN and RF models trained on all samples performed much worse than the global performance metrics indicated. Surprisingly, our variable nearest neighbor method, which utilizes only structurally similar compounds to make predictions, performed reasonably well, suggesting that information of close near neighbors in the training sets is a key determinant of acute toxicity predictions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Liu et al. found that RF and DNN performed better on the global dataset, while k-NN performed better on the unbalanced acute toxicity datasets, using ECFP uniformly.
“It was found that RF and DNN performed better on the global dataset, while k-NN performed better on the unbalanced acute toxicity datasets. This result also highlights the importance of neighbor information in acute toxicity prediction [126].”
From Machine Learning Based Toxicity Prediction: From Chemical Structural Description to Transcriptome Analysis (2018) · cites this paper as [126]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Extended-Connectivity Fingerprints
Rogers, D. O. W., & Hahn, M. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci100050t · Cited by 7,597 · see citing works
Learning from imbalanced data: open challenges and future directions
Krawczyk, B. (2016) · Progress in Artificial Intelligence · doi:10.1007/s13748-016-0094-0 · Cited by 2,459 · see citing works
Theano: A Python framework for fast computation of mathematical expressionsPreprint
Al-Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., Belopolsky, A., et al. (2016) · arXiv.org · doi:10.48550/arxiv.1605.02688 · Cited by 2,438 · see citing works
Molecular graph convolutions: moving beyond fingerprints
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., & Riley, P. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9938-8 · Cited by 1,631 · see citing works
Static Analysis of Shape in TensorFlow ProgramsPreprint
The Theano Development Team, Al‐Rfou, R., Alain, G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,629 · see citing works
Deep Neural Nets as a Method for Quantitative Structure–Activity Relationships
Ma, J., Sheridan, R. P., Liaw, A., Dahl, G. E., & Svetnik, V. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/ci500747n · Cited by 1,159 · see citing works
Duan, J., Dixon, S. L., Lowrie, J. F., & Sherman, W. (2010) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2010.05.008 · Cited by 445 · see citing works
Lenselink, E. B., ten Dijke, N., Bongers, B., Papadatos, G., van Vlijmen, H. W. T., Kowalczyk, W., IJzerman, A. P., & van Westen, G. J. P. (2017) · Journal of Cheminformatics · doi:10.1186/s13321-017-0232-0 · Cited by 334 · see citing works
Koutsoukas, A., Monaghan, K. J., Li, X., & Huan, J. (2017) · Journal of Cheminformatics · doi:10.1186/s13321-017-0226-y · Cited by 315 · see citing works
Toward Good Read-Across Practice (GRAP) guidance
Ball, N. S. (2016) · ALTEX · doi:10.14573/altex.1601251 · Cited by 218 · see citing works
Ball, N. S., Bartels, M. J., Budinsky, R. A., Klapacz, J., Hays, S. M., Kirman, C. R., & Patlewicz, G. Y. (2014) · Regulatory Toxicology and Pharmacology · doi:10.1016/j.yrtph.2013.12.007 · Cited by 58 · see citing works
The acceptance of in silicomodels for REACH: Requirements, barriers, and perspectives
Benfenati, E., Diaza, R. G., Cassano, A., Pardoe, S., Gini, G. C., Mays, C., Knauf, R., & Benighaus, L. (2011) · Chemistry Central Journal · doi:10.1186/1752-153x-5-58 · Cited by 55 · see citing works
Burden, N., Sewell, F. R., & Chapman, K. L. (2015) · PLoS Biology · doi:10.1371/journal.pbio.1002156 · Cited by 53 · see citing works
Yuan, H., Wang, Y., & Cheng, Y. (2006) · Journal of Chemical Information and Modeling · doi:10.1021/ci600299j · Cited by 53 · see citing works
Shao, C.-Y., Chen, S.-Z., Su, B., Tseng, Y. J., Esposito, E. X., & Hopfinger, A. J. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci3005308 · Cited by 45 · see citing works
Liu, R., Tawa, G. J., & Wallqvist, A. (2012) · Chemical Research in Toxicology · doi:10.1021/tx300279f · Cited by 45 · see citing works
Application of Modern Toxicology Approaches for Predicting Acute Toxicity for Chemical Defense
Sciences, E. N. A. O. (2015) · National Academies Press eBooks · doi:10.17226/21775 · Cited by 32 · see citing works
Helgee, E. A., Carlsson, L., Boyer, S. K., & Norinder, U. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci900471e · Cited by 28 · see citing works
Experiences of the REACH testing proposals system to reduce animal testing
Taylor, K. (2014) · ALTEX · doi:10.14573/altex.1311151 · Cited by 25 · see citing works
Sheridan, R. P. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500084w · Cited by 24 · see citing works
The EC White Paper on a Strategy for a Future Chemicals Policy
Vogelgesang, J. (2002) · Alternatives to Laboratory Animals · doi:10.1177/026119290203002s34 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Jiao, Z., Hu, P., Xu, H., & Wang, Q. (2020) · ACS Chemical Health & Safety · doi:10.1021/acs.chas.0c00075 · Cited by 202 · see citing works
Wu, Y., & Wang, G. (2018) · International Journal of Molecular Sciences · doi:10.3390/ijms19082358 · Cited by 193 · see citing works
An Overview of Machine Learning and Big Data for Drug Toxicity Evaluation
Vo, A. H., Vleet, T. R. V., Gupta, R. R., Liguori, M. J., & Rao, M. S. (2019) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.9b00227 · Cited by 191 · see citing works
Role of Artificial Intelligence and Machine Learning in Nanosafety
Winkler, D. A. (2020) · Small · doi:10.1002/smll.202001883 · Cited by 179 · see citing works
Graph attention convolutional neural network model for chemical poisoning of honey bees’ prediction
Wang, F., Yang, J., Wang, M., Jia, C., Shi, X.-X., Hao, G., & Yang, G. (2020) · 中国科学通报:英文版 · doi:10.1016/j.scib.2020.04.006 · Cited by 133 · see citing works
Review of machine learning and deep learning models for toxicity prediction
Guo, W., Liu, J., Dong, F., Song, M., Li, Z., Khan, M. K. H., Patterson, T. A., & Hong, H. (2023) · Experimental Biology and Medicine · doi:10.1177/15353702231209421 · Cited by 104 · see citing works
An Analysis of QSAR Research Based on Machine Learning Concepts
Keyvanpour, M. R., & Shirzad, M. B. (2020) · Current Drug Discovery Technologies · doi:10.2174/1570163817666200316104404 · Cited by 96 · see citing works
Idakwo, G., Thangapandian, S., Luttrell, J., Zhou, Z., Zhang, C., & Gong, P. (2021) · doi:10.21079/11681/41302 · Cited by 82 · see citing works
GGL-Tox: Geometric Graph Learning for Toxicity Prediction
Jiang, J., Wang, R., & Wei, G. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01294 · Cited by 74 · see citing works
Transcriptomics in Toxicogenomics, Part III: Data Modelling for Risk Assessment
Serra, A., Fratello, M., Cattelani, L., Liampa, I., Melagraki, G., Kohonen, P. J., Nymark, P. E. H., Federico, A., Kinaret, P., Jagiełło, K., et al. (2020) · Nanomaterials · doi:10.3390/nano10040708 · Cited by 72 · see citing works
Wang, H., Liu, R., Schyman, P., & Wallqvist, A. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.00042 · Cited by 61 · see citing works
Liu, R., Wang, H., Glover, K., Feasel, M. G., & Wallqvist, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00348 · Cited by 58 · see citing works
Idakwo, G., Thangapandian, S., Luttrell, J., Zhou, Z., Zhang, C., & Gong, P. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.01044 · Cited by 54 · see citing works
Uncertainty-Informed Deep Transfer Learning of Perfluoroalkyl and Polyfluoroalkyl Substance Toxicity
Feinstein, J., Sivaraman, G., Picel, K., Peters, B., Vázquez‐Mayagoitia, Á., Ramanathan, A., MacDonell, M. M., Foster, I. T., & Yan, E. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.1c01204 · Cited by 58 · see citing works
Li, L., Yu, N., Wang, X., Shi, W. F., Liu, H., Zhang, X., Yang, L., Pan, B., Yu, H., & Wei, S. (2022) · Environmental Science & Technology · doi:10.1021/acs.est.2c03345 · Cited by 53 · see citing works
Machine Learning for Environmental Toxicology: A Call for Integration and Innovation
Miller, T. H., Gallidabino, M. D., MacRae, J. I., Högstrand, C., Bury, N. R., Barron, L., Snape, J., & Owen, S. F. (2018) · Environmental Science & Technology · doi:10.1021/acs.est.8b05382 · Cited by 52 · see citing works
TOP: A deep mixture representation learning method for boosting molecular toxicity prediction
Peng, Y., Zhang, Z., Jiang, Q., Guan, J., & Zhou, S. (2020) · Methods · doi:10.1016/j.ymeth.2020.05.013 · Cited by 41 · see citing works
Cheminformatics and artificial intelligence for accelerating agrochemical discovery
Djoumbou-Feunang, Y., Wilmot, J. T., Kinney, J., Chanda, P., Yu, P., Sader, C. A., Sharifi, M., Smith, S., Ou, J., Hu, J. K., et al. (2023) · Frontiers in Chemistry · doi:10.3389/fchem.2023.1292027 · Cited by 41 · see citing works
A comprehensive systematic literature review of ML in nanotechnology for sustainable development
Rehman, I. U., Ullah, I., Khan, H. U., Guellil, M. S., Koo, J., Min, J., Habib, S., Islam, M., & Lee, M. Y. (2024) · Nanotechnology Reviews · doi:10.1515/ntrev-2024-0069 · Cited by 42 · see citing works
Tian, Y., Zhang, S., Yin, H., & Yan, A. (2019) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2019.103888 · Cited by 37 · see citing works
Computational models for predicting liver toxicity in the deep learning era
Mostafa, F. B., & Chen, M. (2024) · Frontiers in Toxicology · doi:10.3389/ftox.2023.1340860 · Cited by 37 · see citing works
Matsuzaka, Y., & Uesawa, Y. (2020) · Molecules · doi:10.3390/molecules25122764 · Cited by 28 · see citing works
Principles and procedures for assessment of acute toxicity incorporating in silico methods
Zwickl, C. M., Graham, J. C., Jolly, R. A., Bassan, A., Ahlberg, E., Amberg, A., Anger, L. T., Beilke, L. D., Bellion, P., Brigo, A., et al. (2022) · Computational Toxicology · doi:10.1016/j.comtox.2022.100237 · Cited by 32 · see citing works
Predicting molecular activity on nuclear receptors by multitask neural networks
Valsecchi, C., Collarile, M., Grisoni, F., Todeschini, R., Ballabio, D., & Consonni, V. (2020) · Journal of Chemometrics · doi:10.1002/cem.3325 · Cited by 26 · see citing works
Krishnan, R., Howard, I. S., Comber, S. D. W., & Jha, A. N. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.164072 · Cited by 24 · see citing works
ToxProfiler: Toxicity-target profiler based on chemical similarity
AbdulHameed, M. D. M., Liu, R., Schyman, P., Sachs, D., Xu, Z., Desai, V., & Wallqvist, A. (2021) · Computational Toxicology · doi:10.1016/j.comtox.2021.100162 · Cited by 22 · see citing works
Hi-MGT: A hybrid molecule graph transformer for toxicity identification
Tan, Z., Zhao, Y., Zhou, T., & Lin, K. (2023) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2023.131808 · Cited by 20 · see citing works
Pereira, V. R., Pereira, D. R., Vieira, K. C. D. M. T., Ribas, V. P., Constantino, C. J. L., Antunes, P. A., & Favareto, A. P. A. (2019) · Environmental Science and Pollution Research · doi:10.1007/s11356-019-06407-0 · Cited by 19 · see citing works
admetSAR—A valuable tool for assisting safety evaluation
Gu, Y., Lou, C., & Tang, Y. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-15339-6.00004-7 · Cited by 18 · see citing works
Multi-principal element nanoparticles: Synthesis strategies and machine learning prediction
Zoubi, W. A., Sheng, Y., Hussain, I., Seongjun, H., & Park, N. (2025) · Coordination Chemistry Reviews · doi:10.1016/j.ccr.2025.216656 · Cited by 18 · see citing works
Jang, Y., Kim, J., Lee, D., Yoo, J. S., Park, J., & Kang, K. (2025) · Computational Toxicology · doi:10.1016/j.comtox.2025.100363 · Cited by 16 · see citing works
TOP: Towards Better Toxicity Prediction by Deep Molecular Representation Learning
Peng, Y., Zhang, Z., Jiang, Q., Guan, J., & Zhou, S. (2019) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm47256.2019.8983340 · Cited by 11 · see citing works
Liu, G., Yuan, M., Chen, X., Lin, X., & Jiang, Q. (2022) · Environmental Science and Pollution Research · doi:10.1007/s11356-022-22943-8 · Cited by 11 · see citing works
Tan, Z., Zhao, Y., Lin, K., & Zhou, T. (2024) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2024.135265 · Cited by 10 · see citing works
AweGNN: Auto-parametrized weighted element-specific graph neural networks for molecules
Szocinski, T., Nguyen, D. D., & Wei, G. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104460 · Cited by 8 · see citing works
Using a Graph Convolutional Neural Network Model to Identify Bile Salt Export Pump Inhibitors
AbdulHameed, M. D. M., Liu, R., & Wallqvist, A. (2023) · ACS Omega · doi:10.1021/acsomega.3c01583 · Cited by 7 · see citing works
Machine Learning in Drug Design
Engkvist, O., Mervin, L., Chen, H., & Ran, T. (2023) · Royal Society of Chemistry eBooks · doi:10.1039/9781788018982-00231 · Cited by 6 · see citing works
Tian, Y., Yang, Z., Wang, H., & Yan, A. (2023) · Chemical Biology & Drug Design · doi:10.1111/cbdd.14214 · Cited by 5 · see citing works
Machine learning based software tools and computational strategies for QSAR modeling
Kaur, R. P., & Manjh, S. (2026) · Discover Chemistry. · doi:10.1007/s44371-026-00663-z · Cited by 3 · see citing works
Rapid screening of chemicals for their potential to cause specific toxidromes
Liu, R., AbdulHameed, M. D. M., Xu, Z., Clancy, B., Desai, V., & Wallqvist, A. (2024) · Frontiers in Drug Discovery · doi:10.3389/fddsv.2024.1324564 · Cited by 3 · see citing works
Meier, M. J., Caiment, F., Corton, J. C., Frötschl, R., Fujita, Y., Jennen, D. G. J., Mezencev, R., & Yauk, C. L. (2024) · Environmental and Molecular Mutagenesis · doi:10.1002/em.22644 · Cited by 3 · see citing works
Eco-genotoxicology: A personal reflection
Jha, A. N. (2025) · Mutation Research/Reviews in Mutation Research · doi:10.1016/j.mrrev.2025.108535 · Cited by 2 · see citing works
Park, M., Shin, Y., Kim, H., & Nam, H. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-01110-4 · Cited by 3 · see citing works
Huang, S., & Hu, M. Y. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-15339-6.00019-9 · Cited by 2 · see citing works
Peng, Y., Zhang, H., Zhang, Z., Lin, Y., Zhou, S., & Qiao, S. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385367 · Cited by 2 · see citing works
Machine Learning (ML) and Toxicity Studies
Ghosh, S., & Dasgupta, R. (2022) · Springer eBooks · doi:10.1007/978-981-16-8881-2_21 · Cited by 1 · see citing works
Uncertainty-Informed Deep Transfer Learning of PFAS ToxicityPreprint
Feinstein, J., Sivaraman, G., Picel, K., Peters, B., Vázquez‐Mayagoitia, Á., Ramanathan, A., MacDonell, M. M., Foster, I. T., & Yan, E. (2021) · ChemRxiv · doi:10.26434/chemrxiv.14397140 · Cited by 1 · see citing works
- Authors
- Liu, R., Madore, M., Glover, K. P., Feasel, M. G., & Wallqvist, A.
- Affiliations
- Telemedicine & Advanced Technology Research Center (US)
- Defense Threat Reduction Agency (US)
- U.S. Army Combat Capabilities Development Command Chemical Biological Center (US)
- Corresponding author
- Liu, Ruifeng
- Wallqvist, Anders
- Venue
- Toxicological Sciences
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-05-02
- Volume
- 164
- Issue
- 2
- Pages
- 512-526
- ISSN
- 1096-0929 (ISSN-L)
- 1096-6080
- OpenAlex
- W2802630768
- Cited by
- 47
- References
- 20
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 5 · 2023: 10 · 2022: 4 · 2021: 6 · 2020: 8 · 2019: 6 · 2018: 3
- FWCI
- 4.29
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Toxicological Sciences (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Army Medical Research
- Defense Threat Reduction Agency (CBCall14-CBS-05-2-0007)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Keywords
- deep neural network, random forest, QSAR, in silico toxicity prediction, alternatives to animal testing
- MeSH terms
- Machine Learning, Deep Learning, Animals, Rabbits, Neural Networks, Computer, Toxicity Tests, Computational Biology, Quantitative Structure-Activity Relationship, Mice, Rats, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review