Beyond the hype: deep neural networks outperform established methods using a ChEMBL bioactivity benchmark set
Open accessPublished version of a preprint
Abstract
The increase of publicly available bioactivity data in recent years has fueled and catalyzed research in chemogenomics, data mining, and modeling approaches. As a direct result, over the past few years a multitude of different methods have been reported and evaluated, such as target fishing, nearest neighbor similarity-based methods, and Quantitative Structure Activity Relationship (QSAR)-based protocols. However, such studies are typically conducted on different datasets, using different validation strategies, and different metrics. In this study, different methods were compared using one single standardized dataset obtained from ChEMBL, which is made available to the public, using standardized metrics (BEDROC and Matthews Correlation Coefficient). Specifically, the performance of Naïve Bayes, Random Forests, Support Vector Machines, Logistic Regression, and Deep Neural Networks was assessed using QSAR and proteochemometric (PCM) methods. All methods were validated using both a random split validation and a temporal validation, with the latter being a more realistic benchmark of expected prospective execution. Deep Neural Networks are the top performing classifiers, highlighting the added value of Deep Neural Networks over other more conventional methods. Moreover, the best method ('DNN_PCM') performed significantly better at almost one standard deviation higher than the mean performance. Furthermore, Multi-task and PCM implementations were shown to improve performance over single task Deep Neural Networks. Conversely, target prediction performed almost two standard deviations under the mean performance. Random Forests, Support Vector Machines, and Logistic Regression performed around mean performance. Finally, using an ensemble of DNNs, alongside additional tuning, enhanced the relative performance by another 27% (compared with unoptimized 'DNN_PCM'). Here, a standardized set to test and evaluate different machine learning algorithms in the context of multi-task learning is offered by providing the data and the protocols. Graphical Abstract .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Comparison of DNNs to other ML methods like RF and XGBoost on standard industrial QSAR datasets shows a statistically significant improvement in prospective predictions.
“Comparison of DNNs to other ML methods like RF and XGBoost on standard industrial QSAR datasets shows a statistically significant improvement in prospective predictions as shown in studies conducted by some of the authors of this paper, and similar conclusions have been published elsewhere.63”
From QSAR without borders (2020) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 1× · see citing works
“We used the MCC and BEDROC as a primary validation metrics [26, 51, 52].”
MethodDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
MoleculeNet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018) · Chemical Science · doi:10.1039/c7sc02664a · Cited by 3,107 · see citing works
Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric
Boughorbel, S., Jarray, F., & El-Anbari, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177678 · Cited by 1,222 · cited 1× · see citing works
“The MCC was calculated to represent the global model quality, and has been shown to be a good metric for unbalanced data-sets [25].”
MethodDeepTox: toxicity prediction using deep learning
Mayr, A., Klambauer, G., Unterthiner, T., & Hochreiter, S. (2016) · Frontiers in Environmental Science · doi:10.3389/fenvs.2015.00080 · Cited by 1,081 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Applications of machine learning in drug discovery and development
Vamathevan, J., Clark, D., Czodrowski, P., Dunham, I., Ferran, E., Lee, G., Li, B., Madabhushi, A., Shah, P., Spitzer, M., et al. (2019) · Nature Reviews Drug Discovery · doi:10.1038/s41573-019-0024-5 · Cited by 3,228 · see citing works
The rise of deep learning in drug discovery
Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., & Blaschke, T. (2018) · Drug Discovery Today · doi:10.1016/j.drudis.2018.01.039 · Cited by 1,770 · see citing works
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063 · see citing works
Muratov, E. N., Bajorath, J., Sheridan, R. P., Tetko, I. V., Filimonov, D., Poroikov, V., Oprea, T. I., Baskin, I. I., Varnek, A., Roitberg, A., et al. (2020) · Chemical Society Reviews · doi:10.1039/d0cs00098a · Cited by 913 · see citing works
Jiang, D., Wu, Z., Hsieh, C.-Y., Chen, G., Liao, B., Wang, Z., Shen, C., Cao, D., Wu, J., & Hou, T. (2021) · Journal of Cheminformatics · doi:10.1186/s13321-020-00479-8 · Cited by 664 · see citing works
Artificial intelligence in COVID-19 drug repurposing
Zhou, Y., Wang, F., Tang, J., Nussinov, R., & Cheng, F. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30192-8 · Cited by 636 · see citing works
Exploiting machine learning for end-to-end drug discovery and development
Ekins, S., Puhl, A. C., Zorn, K. M., Lane, T. R., Russo, D. P., Klein, J. J., Hickey, A. J., & Clark, A. M. (2019) · Nature Materials · doi:10.1038/s41563-019-0338-z · Cited by 578 · see citing works
Bender, A., & Cortés-Ciriano, I. (2021) · Drug Discovery Today · doi:10.1016/j.drudis.2020.12.009 · Cited by 405 · see citing works
Drug–target affinity prediction using graph neural network and contact maps
Jiang, M., Li, Z., Zhang, S., Wang, S., Wang, X., Yuan, Q., & Wei, Z. (2020) · RSC Advances · doi:10.1039/d0ra02297g · Cited by 372 · see citing works
Deep learning in drug discovery: opportunities, challenges and future prospects
Lavecchia, A. (2019) · Drug Discovery Today · doi:10.1016/j.drudis.2019.07.006 · Cited by 353 · see citing works
Artificial intelligence for aging and longevity research: Recent advances and perspectives
Zhavoronkov, A., Mamoshina, P., Vanhaelen, Q., Scheibye-Knudsen, M., Moskalev, A., & Aliper, A. (2019) · Ageing Research Reviews · doi:10.1016/j.arr.2018.11.003 · Cited by 246 · see citing works
Comparing Multiple Machine Learning Algorithms and Metrics for Estrogen Receptor Binding Prediction
Russo, D. P., Zorn, K. M., Clark, A. M., Zhu, H., & Ekins, S. (2018) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.8b00546 · Cited by 152 · see citing works
Amorim, A. M. B., Piochi, L. F., Gaspar, A. T., Preto, A. J., Rosário-Ferreira, N., & Moreira, I. S. (2024) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.3c00352 · Cited by 132 · see citing works
Coloring Molecules with Explainable Artificial Intelligence for Preclinical Relevance Assessment
Jiménez-Luna, J., Skalic, M., Weskamp, N., & Schneider, G. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01344 · Cited by 96 · see citing works
Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
Cortés-Ciriano, I., & Bender, A. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00297 · Cited by 51 · see citing works
Proteochemometrics – recent developments in bioactivity and selectivity modeling
Bongers, B. J., IJzerman, A. P., & Van Westen, G. J. P. (2019) · Drug Discovery Today: Technologies · doi:10.1016/j.ddtec.2020.08.003 · Cited by 54 · see citing works
Self-citationAssessing Deep and Shallow Learning Methods for Quantitative Prediction of Acute Chemical Toxicity
Liu, R., Madore, M., Glover, K. P., Feasel, M. G., & Wallqvist, A. (2018) · Toxicological Sciences · doi:10.1093/toxsci/kfy111 · Cited by 47 · see citing works
Gellatly, N., & Sewell, F. (2019) · Computational Toxicology · doi:10.1016/j.comtox.2019.03.003 · Cited by 43 · see citing works
- Authors
- Lenselink, E. B., ten Dijke, N., Bongers, B., Papadatos, G., van Vlijmen, H. W. T., Kowalczyk, W., IJzerman, A. P., & van Westen, G. J. P.
- Affiliations
- Leiden University (NL)
- European Bioinformatics Institute (GB)
- ORCID
- Venue
- Journal of Cheminformatics
- Published
- 2017-08-14
- Volume
- 9
- Issue
- 1
- Pages
- 45-45
- ISSN
- 1758-2946 (ISSN-L)
- PMC
- PMC5555960
- OpenAlex
- W2740946158
- Cited by
- 334
- References
- 5
- Citations by year
- 2026: 18 · 2025: 35 · 2024: 41 · 2023: 34 · 2022: 40 · 2021: 49 · 2020: 39 · 2019: 45 · 2018: 32 · 2017: 1
- FWCI
- 24.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Cheminformatics (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- Funders
- NWO (NWO-TOP#714.011.001)
- NWO-STW (STW-Veni#14410)
- EMBL-Memberstates
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Keywords
- deep neural network, ChEMBL, bioactivity prediction, QSAR, proteochemometrics, multi-task learning, ensemble learning, target prediction, temporal validation, Matthews correlation coefficient, chemogenomics, benchmark dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
