The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
In Need of Bias Control: Evaluating Chemical Data for Machine Learning in Structure-Based Virtual Screening
Abstract
Reports of successful applications of machine learning (ML) methods in structure-based virtual screening (SBVS) are increasing. ML methods such as convolutional neural networks show promising results and often outperform traditional methods such as empirical scoring functions in retrospective validation. However, trained ML models are often treated as black boxes and are not straightforwardly interpretable. In most cases, it is unknown which features in the data are decisive and whether a model's predictions are right for the right reason. Hence, we re-evaluated three widely used benchmark data sets in the context of ML methods and came to the conclusion that not every benchmark data set is suitable. Moreover, we demonstrate on two examples from current literature that bias is learned implicitly and unnoticed from standard benchmarks. On the basis of these results, we conclude that there is a need for eligible validation experiments and benchmark data sets suited to ML for more bias-controlled validation in ML-based SBVS. Therefore, we provide guidelines for setting up validation experiments and give a perspective on how new data sets could be generated.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
ChEMBL: a large-scale bioactivity database for drug discovery
Gaulton, A., Bellis, L. J., Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y., McGlinchey, S., Michalovich, D., Al-Lazikani, B., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr777 · Cited by 4,382
ZINC − A Free Database of Commercially Available Compounds for Virtual Screening
Irwin, J. J., & Shoichet, B. K. (2005) · Journal of Chemical Information and Modeling · doi:10.1021/ci049714+ · Cited by 4,058
Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling
Svetnik, V., Liaw, A., Tong, C., Culberson, J. C., Sheridan, R. P., & Feuston, B. P. (2003) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci034160g · Cited by 3,573
MoleculeNet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018) · Chemical Science · doi:10.1039/c7sc02664a · Cited by 3,107
The Properties of Known Drugs. 1. Molecular Frameworks
Bemis, G. W., & Murcko, M. A. (1996) · Journal of Medicinal Chemistry · doi:10.1021/jm9602928 · Cited by 2,482
Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking.
Mysinger, M. M., Carchia, M., Irwin, J. J., & Shoichet, B. K. (2012) · Journal of Medicinal Chemistry · doi:10.1021/jm300687e · Cited by 2,412
Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17
Ruddigkeit, L., van Deursen, R., Blum, L. C., & Reymond, J.-L. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci300415d · Cited by 1,505
Benchmarking sets for molecular docking.
Huang, N., Shoichet, B. K., & Irwin, J. J. (2006) · Journal of Medicinal Chemistry · doi:10.1021/jm0608356 · Cited by 1,318
Database resources of the National Center for Biotechnology Information
Wheeler, D. L., Barrett, T., Benson, D. A., Bryant, S. H., Canese, K., Chetvernin, V., Church, D. M., DiCuccio, M., Edgar, R., Federhen, S., et al. (2007) · Nucleic Acids Research · doi:10.1093/nar/gkm1000 · Cited by 1,007
Ballester, P. J., & Mitchell, J. B. O. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq112 · Cited by 989
KDEEP: Protein–Ligand Absolute Binding Affinity Prediction via 3D-Convolutional Neural Networks
Jiménez, J., Škalič, M., Martínez-Rosell, G., & De Fabritiis, G. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00650 · Cited by 926
Protein–Ligand Scoring with Convolutional Neural Networks
Ragoza, M., Hochuli, J., Idrobo, E., Sunseri, J., & Koes, D. R. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00740 · Cited by 774
Bissantz, C., Folkers, G., & Rognan, D. (2000) · Journal of Medicinal Chemistry · doi:10.1021/jm001044l · Cited by 740
Forging the basis for developing protein-ligand interaction scoring functions.
Liu, Z., Su, M., Han, L., Liu, J., Yang, Q., Li, Y., & Wang, R. (2017) · Accounts of Chemical Research · doi:10.1021/acs.accounts.6b00491 · Cited by 493
Maximum Unbiased Validation (MUV) Data Sets for Virtual Screening Based on PubChem Bioactivity Data
Rohrer, S. G., & Baumann, K. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci8002649 · Cited by 402
Boosting docking-based virtual screening with deep learning.
Pereira, J. C., Caffarena, E. R., & dos Santos, C. N. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00355 · Cited by 395
Virtual Screening Using Protein−Ligand Docking: Avoiding Artificial Enrichment
Verdonk, M. L., Berdini, V., Hartshorn, M. J., Mooij, W. T. M., Murray, C. W., Taylor, R. D., & Watson, P. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci034289q · Cited by 380
Docking and scoring with ICM: the benchmarking results and strategies for improvement
Neves, M. A. C., Totrov, M., & Abagyan, R. (2012) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-012-9547-0 · Cited by 371
Performance of machine-learning scoring functions in structure-based virtual screening.
Wójcikowski, M., Ballester, P. J., & Siedlecki, P. (2017) · Scientific Reports · doi:10.1038/srep46710 · Cited by 364
Open-source platform to benchmark fingerprints for ligand-based virtual screening
Riniker, S., & Landrum, G. A. (2013) · Journal of Cheminformatics · doi:10.1186/1758-2946-5-26 · Cited by 359
Structure-Based Discovery of Inhibitors of Thymidylate Synthase
Shoichet, B. K., Stroud, R. M., Santi, D. V., Kuntz, I. D., & Perry, K. M. (1993) · Science · doi:10.1126/science.8451640 · Cited by 322
Rapid Context-Dependent Ligand Desolvation in Molecular Docking
Mysinger, M. M., & Shoichet, B. K. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci100214a · Cited by 317
Classification of Current Scoring Functions
Liu, J., & Wang, R. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/ci500731a · Cited by 303
McGovern, S. L., & Shoichet, B. K. (2003) · Journal of Medicinal Chemistry · doi:10.1021/jm0300330 · Cited by 276
Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization
Wallach, I., & Heifets, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00403 · Cited by 260
Interpretation of Quantitative Structure-Activity Relationship Models: Past, Present, and Future
Polishchuk, P. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00274 · Cited by 236
Bauer, M. R., Ibrahim, T. M., Vogel, S. M., & Boeckler, F. M. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400115b · Cited by 200
Gaieb, Z., Liu, S., Gathiaka, S., Chiu, M., Yang, H., Shao, C., Feher, V. A., Walters, W. P., Kuhn, B., Rudolph, M. G., et al. (2018) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-017-0088-4 · Cited by 193
Good, A. C., & Oprea, T. I. (2008) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9167-2 · Cited by 180
CSAR Benchmark Exercise of 2010: Selection of the Protein–Ligand Complexes
Dunbar, J. B., Smith, R. D., Yang, C.-Y., Ung, P. M.-U., Lexa, K. W., Khazanov, N. A., Stuckey, J. A., Wang, S., & Carlson, H. A. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci200082t · Cited by 175
Comment on “Predicting reaction performance in C–N cross-coupling using machine learning”
Chuang, K. V., & Keiser, M. J. (2018) · Science · doi:10.1126/science.aat8603 · Cited by 148
Ihlenfeldt, W. D., TAKAHASHI, Y., Abe, H., & Sasaki, S. (1994) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci00017a013 · Cited by 144
Beware of machine learning-based scoring functions-on the danger of developing black boxes.
Gabel, J., Desaphy, J., & Rognan, D. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500406k · Cited by 143
Kinases, Homology Models, and High Throughput Docking
Diller, D., & Li, R. (2003) · Journal of Medicinal Chemistry · doi:10.1021/jm020503a · Cited by 120
Decoys Selection in Benchmarking Datasets: Overview and Perspectives
Réau, M. F., Langenfeld, F., Zagury, J., Lagarde, N., & Montès, M. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.00011 · Cited by 118
Armstrong, M. S., Morris, G. M., Finn, P. W., Sharma, R., Moretti, L., Cooper, R. I., & Richards, W. G. (2010) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-010-9374-0 · Cited by 108
CSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma
Carlson, H. A., Smith, R. D., Damm‐Ganamet, K. L., Stuckey, J. A., Ahmed, A., Convery, M. A., Somers, D. O., Kranz, M., Elkins, P. A., Cui, G., et al. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00523 · Cited by 105
Vogel, S., Bauer, M. R., & Boeckler, F. M. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci2001549 · Cited by 97
Cleves, A. E., & Jain, A. N. (2007) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9150-y · Cited by 73
Virtual Decoy Sets for Molecular Docking Benchmarks
Wallach, I., & Lilien, R. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci100374f · Cited by 60
Benchmarking methods and data sets for ligand enrichment assessment in virtual screening
Xia, J., Tilahun, E. L., Reid, T.-E. R., Zhang, L., & Wang, X. S. (2014) · Methods · doi:10.1016/j.ymeth.2014.11.015 · Cited by 51
Does your model weigh the same as a Duck?
Jain, A. N., & Cleves, A. E. (2011) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-011-9530-1 · Cited by 26
Discriminative Chemical Patterns: Automatic and Interactive Design
Bietz, S., Schomburg, K. T., Hilbig, M., & Rarey, M. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00323 · Cited by 12
Self-citationMaximal Unbiased Benchmarking Data Sets for Human Chemokine Receptors and Comparative Analysis
Xia, J., Reid, T.-E. R., Wu, S., Zhang, L., & Wang, X. S. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00004 · Cited by 5
The art and practice of structure-based drug design: A molecular modeling perspective
(1996) · doi:10.1002/(sici)1098-1128(199601)16:1<3::aid-med1>3.0.co;2-6
Cited by
Key Topics in Molecular Docking for Drug Design
Torres, P. H. M., Sodero, A. C. R., Jofily, P., & Silva-Jr, F. P. (2019) · International Journal of Molecular Sciences · doi:10.3390/ijms20184574 · Cited by 641
Chen, L., Tan, X., Wang, D., Zhong, F., Liu, X., Yang, T., Luo, X., Chen, K., Jiang, H., & Zheng, M. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa524 · Cited by 607
Rodríguez-Pérez, R., & Bajorath, J. (2020) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.9b01101 · Cited by 512
Artificial intelligence in drug discovery: recent advances and future perspectives
Jiménez-Luna, J., Grisoni, F., Weskamp, N., & Schneider, G. (2021) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2021.1909567 · Cited by 464
Artificial intelligence for natural product drug discovery
Mullowney, M. W., Duncan, K. R., Elsayed, S. S., Garg, N., van der Hooft, J. J. J., Martin, N. I., Meijer, D., Terlouw, B. R., Biermann, F., Blin, K., et al. (2023) · Nature Reviews Drug Discovery · doi:10.1038/s41573-023-00774-7 · Cited by 460
Metadata
- Authors
- Sieg, J., Flachsenberg, F., & Rarey, M.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2019-03-05
- Volume
- 59
- Issue
- 3
- Pages
- 947-961
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2918239264
- Cited by
- 321
- References
- 49
- FWCI
- 25.70
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.