Rational selection of training and test sets for the development of validated QSAR models
Abstract
Quantitative Structure-Activity Relationship (QSAR) models are used increasingly to screen chemical databases and/or virtual chemical libraries for potentially bioactive molecules. These developments emphasize the importance of rigorous model validation to ensure that the models have acceptable predictive power. Using k nearest neighbors (kNN) variable selection QSAR method for the analysis of several datasets, we have demonstrated recently that the widely accepted leave-one-out (LOO) cross-validated R2 (q2) is an inadequate characteristic to assess the predictive ability of the models [Golbraikh, A., Tropsha, A. Beware of q2! J. Mol. Graphics Mod. 20, 269-276, (2002)]. Herein, we provide additional evidence that there exists no correlation between the values of q2 for the training set and accuracy of prediction (R2) for the test set and argue that this observation is a general property of any QSAR model developed with LOO cross-validation. We suggest that external validation using rationally selected training and test sets provides a means to establish a reliable QSAR model. We propose several approaches to the division of experimental datasets into training and test sets and apply them in QSAR studies of 48 functionalized amino acid anticonvulsants and a series of 157 epipodophyllotoxin derivatives with antitumor activity. We formulate a set of general criteria for the evaluation of predictive power of QSAR models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Golbraikh et al. proposed a protocol to determine if a QSAR model has predictive power.
“Golbraikh et al. 43 proposed a protocol to determine if a QSAR model has a predictive power.”
From Quantitative Toxicity Prediction Using Topology Based Multitask Deep Neural Networks (2018) · cites this paper as [43]
Cites
Golbraikh, A., & Tropsha, A. (2002) · Journal of Molecular Graphics and Modelling · doi:10.1016/s1093-3263(01)00123-1 · Cited by 3,735 · see citing works
Self-citationSachs, L. (1984) · Springer series in statistics · doi:10.1007/978-1-4612-5246-7 · Cited by 1,016 · see citing works
Chemometric methods in molecular design
Waterbeemd, H. V. D. (1995) · Medical Entomology and Zoology · Cited by 561 · see citing works
Golbraikh, A., & Tropsha, A. (2002) · Journal of Computer-Aided Molecular Design · doi:10.1023/a:1020869118689 · Cited by 504 · see citing works
Self-citationZheng, W., & Tropsha, A. (1999) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci980033m · Cited by 450 · see citing works
Self-citationChemometric Methods in Molecular Design
(1995) · Methods and principles in medicinal chemistry · doi:10.1002/9783527615452 · Cited by 330 · see citing works
Kubinyi, H., Hamprecht, F. A., & Mietzner, T. (1998) · Journal of Medicinal Chemistry · doi:10.1021/jm970732a · Cited by 242 · see citing works
Comparison of algorithms for dissimilarity-based compound selection
Snarey, M., Terrett, N. K., Willett, P., et al. (1997) · Journal of Molecular Graphics and Modelling · doi:10.1016/s1093-3263(98)00008-4 · Cited by 205 · see citing works
Fan, Y., Shi, L. M., Kohn, K. W., Pommier, Y. G., & Weinstein, J. N. (2001) · Journal of Medicinal Chemistry · doi:10.1021/jm0005151 · Cited by 162 · see citing works
Recanatini, M., Cavalli, A., Belluti, F., Piazzi, L., Rampa, A., Bisi, A., Gobbi, S., Valenti, P., Andrisano, V., Bartolini, M., et al. (2000) · Journal of Medicinal Chemistry · doi:10.1021/jm990971t · Cited by 154 · see citing works
Evolutionary variable selection in regression and PLS analyses
Kubinyi, H. (1996) · Journal of Chemometrics · doi:10.1002/(sici)1099-128x(199603)10:2 · Cited by 152 · see citing works
Shen, M., LeTiran, A., Xiao, Y.-D., Golbraikh, A., Kohn, H., & Tropsha, A. (2002) · Journal of Medicinal Chemistry · doi:10.1021/jm010488u · Cited by 145 · see citing works
Self-citationXiao, Z., Xiao, Y.-D., Feng, J., Golbraikh, A., Tropsha, A., & Lee, K. (2002) · Journal of Medicinal Chemistry · doi:10.1021/jm0105427 · Cited by 90 · see citing works
Self-citationCho, S. J., Zheng, W., & Tropsha, A. (1998) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci9700945 · Cited by 83 · see citing works
Self-citationCho, S. J., Tropsha, A., Suffness, M., et al. (1996) · Journal of Medicinal Chemistry · doi:10.1021/jm9503052 · Cited by 81 · see citing works
Self-citationHoffman, B. T., Cho, S. J., Zheng, W., Wyrick, S. D., Nichols, D. E., Mailman, R. B., & Tropsha, A. (1999) · Journal of Medicinal Chemistry · doi:10.1021/jm980415j · Cited by 79 · see citing works
Self-citationMorón, J. A., Campillo, M., Pérez, V., Unzeta, M., & Pardo, L. (2000) · Journal of Medicinal Chemistry · doi:10.1021/jm991164x · Cited by 66 · see citing works
Tropsha, A., & Weifan, Z. (2001) · Current Pharmaceutical Design · doi:10.2174/1381612013397834 · Cited by 61 · see citing works
Self-citationA Unified Framework for Using Neural Networks To Build QSARs
Ajay (1993) · Journal of Medicinal Chemistry · doi:10.1021/jm00075a011 · Cited by 54 · see citing works
Gironés, X., Gallegos, A., & Carbó‐Dorca, R. (2000) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci0004558 · Cited by 51 · see citing works
Reynolds, C. H., Druker, R., & Pfahler, L. B. (1998) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci970056l · Cited by 49 · see citing works
Use of comparative molecular field analysis and cluster analysis in series design
Novellino, E., Fattorusso, C., & Greco, G. (1995) · Pharmaceutica Acta Helvetiae · doi:10.1016/0031-6865(95)00014-z · Cited by 43 · see citing works
Suzuki, T., Ide, K., Ishida, M., & Shapiro, S. (2001) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci000333f · Cited by 42 · see citing works
Golbraikh, A. (2000) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci990437u · Cited by 37 · see citing works
Self-citationConstruction of High-Quality Structure−Property−Activity Regressions: The Boiling Points of Sulfides
Randić, M., & Basak, S. C. (2000) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci990115q · Cited by 30 · see citing works
Bordás, B., Kőmíves, T., Szántó, Z., et al. (2000) · Journal of Agricultural and Food Chemistry · doi:10.1021/jf990395+ · Cited by 26 · see citing works
Tropsha, A., Cho, S.-J., & Zheng, W. (1999) · ACS symposium series · doi:10.1021/bk-1999-0719.ch013 · Cited by 22 · see citing works
Self-citationXie, D., Tropsha, A., & Schlick, T. (1999) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci990333j · Cited by 20 · see citing works
Self-citationGussio, R., Pattabiraman, N., Kellogg, G. E., et al. (1998) · Methods · doi:10.1006/meth.1998.0582 · Cited by 18 · see citing works
[Modeling of a three-dimensional structure of cytochrome P-450 1A2 and search for its new ligands].
Belkina, N. V., Skvortsov, V. S., Ivanov, A. S., et al. (1999) · PubMed · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Quantitative Structure−Activity Relationship Modeling of Rat Acute Toxicity by Oral Exposure
Zhu, H., Martin, T. M., Ye, L., Sedykh, A., Young, D. M., & Tropsha, A. (2009) · Chemical Research in Toxicology · doi:10.1021/tx900189p · Cited by 287 · see citing works
Self-citationQuantitative Toxicity Prediction Using Topology Based Multitask Deep Neural Networks
Wu, K., & Wei, G.-W. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00558 · Cited by 160 · see citing works
An automated PLS search for biologically relevant QSAR descriptors
Olah, M., Bologa, C., & Oprea, T. I. (2004) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-004-4060-8 · Cited by 91 · see citing works
Cheminformatics-driven discovery of polymeric micelle formulations for poorly soluble drugs
Alves, V. M., Hwang, D., Muratov, E., Sokolsky-Papkov, M., Varlamova, E., Vinod, N., Lim, C., Andrade, C. H., Tropsha, A., & Kabanov, A. (2019) · Science Advances · doi:10.1126/sciadv.aav9784 · Cited by 56 · see citing works
Self-citationLuo, M., Wang, X. S., Roth, B. L., Golbraikh, A., & Tropsha, A. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci400460q · Cited by 37 · see citing works
Self-citation
- Authors
- Golbraikh, A., Shen, M., Xiao, Z., Xiao, Y.-D., Lee, K.-H., & Tropsha, A.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Corresponding author
- Tropsha, Alexander
- Venue
- Journal of Computer-Aided Molecular Design
- Published
- 2003-02
- Volume
- 17
- Issue
- 2-4
- Pages
- 241-253
- ISSN
- 0920-654X (ISSN-L)
- 1573-4951
- OpenAlex
- W1571439140
- Cited by
- 712
- References
- 30
- Citations by year
- 2026: 24 · 2025: 36 · 2024: 26 · 2023: 41 · 2022: 35 · 2021: 44 · 2020: 47 · 2019: 38 · 2018: 36 · 2017: 31 · 2016: 35 · 2015: 36 · 2014: 28 · 2013: 31 · 2012: 30
- FWCI
- 13.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Journal of Computer-Aided Molecular Design (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (5r03mh060328-02)
- National Institute of Mental Health (MH60328)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Advanced Chemical Physics Studies
- Keywords
- QSAR model validation, leave-one-out cross-validation, external validation, epipodophyllotoxin derivatives, anticancer activity, virtual screening
- MeSH terms
- Algorithms, Anticonvulsants, Models, Chemical, Software Validation, Molecular Structure, Databases, Factual, Quantitative Structure-Activity Relationship
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
