The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization
Open accessAbstract
Undetected overfitting can occur when there are significant redundancies between training and validation data. We describe AVE, a new measure of training-validation redundancy for ligand-based classification problems, that accounts for the similarity among inactive molecules as well as active ones. We investigated seven widely used benchmarks for virtual screening and classification, and we show that the amount of AVE bias strongly correlates with the performance of ligand-based predictive methods irrespective of the predicted property, chemical fingerprint, similarity measure, or previously applied unbiasing techniques. Therefore, it may be the case that the previously reported performance of most ligand-based methods can be explained by overfitting to benchmarks rather than good prospective accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Extended-Connectivity Fingerprints
Rogers, D., & Hahn, M. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci100050t · Cited by 7,524
The Properties of Known Drugs. 1. Molecular Frameworks
Bemis, G. W., & Murcko, M. A. (1996) · Journal of Medicinal Chemistry · doi:10.1021/jm9602928 · Cited by 2,482
Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking.
Mysinger, M. M., Carchia, M., Irwin, J. J., & Shoichet, B. K. (2012) · Journal of Medicinal Chemistry · doi:10.1021/jm300687e · Cited by 2,412
Algorithms for hierarchical clustering: an overview
Murtagh, F., & Contreras, P. (2012) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.53 · Cited by 1,874
Reoptimization of MDL Keys for Use in Drug Discovery
Durant, J. L., Leland, B. A., Henry, D. R., & Nourse, J. G. (2002) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci010132r · Cited by 1,862
Molecular graph convolutions: moving beyond fingerprints
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., & Riley, P. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9938-8 · Cited by 1,631
Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17
Ruddigkeit, L., van Deursen, R., Blum, L. C., & Reymond, J.-L. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci300415d · Cited by 1,505
The ChEMBL bioactivity database: an update
Bento, A. P., Gaulton, A., Hersey, A., Bellis, L. J., Chambers, J., Davies, M., Krüger, F. A., Light, Y., Mak, L., McGlinchey, S., et al. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1031 · Cited by 1,492
The SIDER database of drugs and side effects
Kuhn, M., Letunic, I., Jensen, L. J., & Bork, P. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1075 · Cited by 1,451
Deep Neural Nets as a Method for Quantitative Structure–Activity Relationships
Ma, J., Sheridan, R. P., Liaw, A., Dahl, G. E., & Svetnik, V. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/ci500747n · Cited by 1,159
Profound Methyl Effects in Drug Discovery and a Call for New CH Methylation Reactions
Schönherr, H., & Cernak, T. (2013) · Angewandte Chemie International Edition · doi:10.1002/anie.201303207 · Cited by 1,094
Atom pairs as molecular features in structure-activity studies: definition and applications
Carhart, R. E., Smith, D. H., & Venkataraghavan, R. (1985) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci00046a002 · Cited by 1,066
Large-scale prediction and testing of drug activity on side-effect targets
Lounkine, E., Keiser, M. J., Whitebread, S., Mikhailov, D., Hamon, J., Jenkins, J. L., Lavan, P., Weber, E., Doak, A. K., Côté, S., et al. (2012) · Nature · doi:10.1038/nature11159 · Cited by 881
Wang, Y., Xiao, J., Suzek, T. O., Zhang, J., Wang, J., Zhou, Z., Han, L., Karapetyan, K., Dracheva, S., Shoemaker, B. A., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr1132 · Cited by 659
Maximum Unbiased Validation (MUV) Data Sets for Virtual Screening Based on PubChem Bioactivity Data
Rohrer, S. G., & Baumann, K. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci8002649 · Cited by 402
Virtual Screening Using Protein−Ligand Docking: Avoiding Artificial Enrichment
Verdonk, M. L., Berdini, V., Hartshorn, M. J., Mooij, W. T. M., Murray, C. W., Taylor, R. D., & Watson, P. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci034289q · Cited by 380
Comparability of Mixed IC50 Data – A Statistical Analysis
Kalliokoski, T., Kramer, C., Vulpetti, A., & Gedeck, P. (2013) · PLoS ONE · doi:10.1371/journal.pone.0061007 · Cited by 315
Chemical Similarity Using Physiochemical Property Descriptors
Kearsley, S. K., Sallamack, S., Fluder, E. M., Andose, J. D., Mosley, R. T., & Sheridan, R. P. (1996) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci950274j · Cited by 267
A Global Map of Lipid-Binding Proteins and Their Ligandability in Cells
Niphakis, M. J., Lum, K. M., Cognetta, A. B., Correia, B. E., Ichu, T.-A., Olucha, J., Brown, S. J., Kundu, S., Piscitelli, F., Rosen, H., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.045 · Cited by 239
The Experimental Uncertainty of Heterogeneous Public Ki Data
Kramer, C., Kalliokoski, T., Gedeck, P., & Vulpetti, A. (2012) · Journal of Medicinal Chemistry · doi:10.1021/jm300131x · Cited by 233
MMP-Cliffs: Systematic Identification of Activity Cliffs on the Basis of Matched Molecular Pairs
Hu, X., Hu, Y., Vogt, M., Stumpfe, D., & Bajorath, J. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci3001138 · Cited by 203
Good, A. C., & Oprea, T. I. (2008) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9167-2 · Cited by 180
Bias, reporting, and sharing: computational evaluations of docking methods
Jain, A. N. (2008) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9151-x · Cited by 159
Martin, É., Polyakov, V. R., Tian, L., & Perez, R. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00166 · Cited by 81
Cleves, A. E., & Jain, A. N. (2007) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9150-y · Cited by 73
Chemical Structural Novelty: On-Targets and Off-Targets
Yera, E. R., Cleves, A. E., & Jain, A. N. (2011) · Journal of Medicinal Chemistry · doi:10.1021/jm200666a · Cited by 62
Xia, J., Jin, H., Liu, Z., Zhang, L., & Wang, X. S. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500062f · Cited by 57
Systematic Identification and Classification of Three-Dimensional Activity Cliffs
Hu, Y., Furtmann, N., Gütschow, M., & Bajorath, J. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci300158v · Cited by 33
Does your model weigh the same as a Duck?
Jain, A. N., & Cleves, A. E. (2011) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-011-9530-1 · Cited by 26
Ripphausen, P., Wassermann, A. M., & Bajorath, J. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci200309j · Cited by 21
Electrostatic evaluation of isosteric analogues
Sayle, R. A., & Nicholls, A. O. (2006) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-006-9045-3 · Cited by 8
Cited by
Sieg, J., Flachsenberg, F., & Rarey, M. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00712 · Cited by 321
Synthetic organic chemistry driven by artificial intelligence
de Almeida, A. F., Moreira, R., & Rodrigues, T. (2019) · Nature Reviews Chemistry · doi:10.1038/s41570-019-0124-0 · Cited by 303
Chen, L., Cruz, A., Ramsey, S., Dickson, C. J., Duca, J. S., Hornak, V., Koes, D. R., & Kurtzman, T. (2019) · PLOS ONE · doi:10.1371/journal.pone.0220113 · Cited by 283
LIT-PCBA: An Unbiased Data Set for Machine Learning and Virtual Screening
Tran-Nguyen, V.-K., Jacquemard, C., & Rognan, D. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00155 · Cited by 259
Janet, J. P., Ramesh, S., Duan, C., & Kulik, H. J. (2020) · ACS Central Science · doi:10.1021/acscentsci.0c00026 · Cited by 200
Metadata
- Authors
- Wallach, I., & Heifets, A.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2018-04-26
- Volume
- 58
- Issue
- 5
- Pages
- 916-932
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2803094965
- Cited by
- 260
- References
- 32
- FWCI
- 14.03
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.