Time-Split Cross-Validation as a Method for Estimating the Goodness of Prospective Prediction.
Abstract
Cross-validation is a common method to validate a QSAR model. In cross-validation, some compounds are held out as a test set, while the remaining compounds form a training set. A model is built from the training set, and the test set compounds are predicted on that model. The agreement of the predicted and observed activity values of the test set (measured by, say, R(2)) is an estimate of the self-consistency of the model and is sometimes taken as an indication of the predictivity of the model. This estimate of predictivity can be optimistic or pessimistic compared to true prospective prediction, depending how compounds in the test set are selected. Here, we show that time-split selection gives an R(2) that is more like that of true prospective prediction than the R(2) from random selection (too optimistic) or from our analog of leave-class-out selection (too pessimistic). Time-split selection should be used in addition to random selection as a standard for cross-validation in QSAR model building.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A time-split test set better simulates the natural evolution of a typical drug discovery project.
“A better ways to simulate the natural evolution of a typical drug discovery project is to use a time-split test set,31 i.e., assigning compounds tested in later phases of the project to the test set.”
From QSAR without borders (2020) · cites this paper as [31]
Time-split gives a good estimate of the R2 for true prospective prediction relative to random test set selection, which can overestimate prediction accuracy, and leave-class-out validation, which is too pessimistic.
“It can be demonstrated that time-split gives a good estimate of the R2 for true prospective prediction relative to random test set selection (a standard method that can overestimate prediction accuracy) and leave-class-out validation (which is too pessimistic).31”
From QSAR without borders (2020) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Golbraikh, A., & Tropsha, A. (2002) · Journal of Molecular Graphics and Modelling · doi:10.1016/s1093-3263(01)00123-1 · Cited by 3,735 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
MoleculeNet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018) · Chemical Science · doi:10.1039/c7sc02664a · Cited by 3,107 · see citing works
QSAR Modeling: Where have you been? Where are you going to?
Cherkasov, A., Muratov, E. N., Fourches, D., Varnek, A., Baskin, I. I., Cronin, M., Dearden, J., Gramatica, P., Martin, Y. C., Todeschini, R., et al. (2014) · Journal of Medicinal Chemistry · doi:10.1021/jm4004285 · Cited by 2,164 · see citing works
Analyzing Learned Molecular Representations for Property Prediction
Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00237 · Cited by 1,923 · see citing works
Planning chemical syntheses with deep neural networks and symbolic AI
Segler, M. H. S., Preuss, M., & Waller, M. P. (2018) · Nature · doi:10.1038/nature25978 · Cited by 1,847 · see citing works
Muratov, E. N., Bajorath, J., Sheridan, R. P., Tetko, I. V., Filimonov, D., Poroikov, V., Oprea, T. I., Baskin, I. I., Varnek, A., Roitberg, A., et al. (2020) · Chemical Society Reviews · doi:10.1039/d0cs00098a · Cited by 913 · see citing works
Self-citationLarge-scale comparison of machine learning methods for drug target prediction on ChEMBL
Mayr, A., Klambauer, G., Unterthiner, T., Steijaert, M., Wegner, J. K., Ceulemans, H., Clevert, D.-A., & Hochreiter, S. (2018) · Chemical Science · doi:10.1039/c8sc00148k · Cited by 584 · see citing works
Extreme Gradient Boosting as a Method for Quantitative Structure–Activity Relationships
Sheridan, R. P., Wang, W. M., Liaw, A., Ma, J., & Gifford, E. M. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00591 · Cited by 544 · see citing works
Self-citationOpportunities and Challenges for Machine Learning in Materials Science
Morgan, D., & Jacobs, R. (2020) · Annual Review of Materials Research · doi:10.1146/annurev-matsci-070218-010015 · Cited by 549 · see citing works
Convolutional embedding of attributed molecular graphs for physical property prediction
Coley, C. W., Barzilay, R., Green, W. H., Jaakkola, T. S., & Jensen, K. F. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00601 · Cited by 470 · see citing works
Is Multitask Deep Learning Practical for Pharma?
Ramsundar, B., Liu, B., Wu, Z., Verras, A., Tudor, M., Sheridan, R. P., & Pande, V. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00146 · Cited by 263 · see citing works
Self-citationDemystifying Multitask Deep Neural Networks for Quantitative Structure–Activity Relationships
Xu, Y., Ma, J., Liaw, A., Sheridan, R. P., & Svetnik, V. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00087 · Cited by 238 · see citing works
Self-citationThree pitfalls to avoid in machine learning.
Riley, P. (2019) · Nature · doi:10.1038/d41586-019-02307-y · Cited by 142 · see citing works
Sheridan, R. P. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00825 · Cited by 84 · see citing works
Self-citationAdding Stochastic Negative Examples into Machine Learning Improves Molecular Bioactivity Prediction
Cáceres, E. L., Mew, N. C., & Keiser, M. J. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00565 · Cited by 33 · see citing works
- Authors
- Sheridan, R. P.
- Affiliations
- Merck & Co., Inc., Rahway, NJ, USA (United States) (US)
- Corresponding author
- Sheridan, Robert P.
- ORCID
- Sheridan, Robert P.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2013-04-05
- Volume
- 53
- Issue
- 4
- Pages
- 783-790
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2019678805
- Cited by
- 376
- References
- 1
- Citations by year
- 2026: 96 · 2025: 31 · 2024: 34 · 2023: 34 · 2022: 23 · 2021: 41 · 2020: 26 · 2019: 30 · 2018: 20 · 2017: 15 · 2016: 10 · 2015: 7 · 2014: 4 · 2013: 5
- FWCI
- 9.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Chemical Information and Modeling (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Physical Sciences › Chemistry › Analytical Chemistry › Spectroscopy and Chemometric Analyses
- Keywords
- QSAR model validation, prospective prediction, test set selection, activity prediction
- MeSH terms
- Algorithms, Binding Sites, Computer Simulation, Humans, Ligands, Predictive Value of Tests, Protein Binding, Proteins, Time Factors, Models, Statistical, Quantitative Structure-Activity Relationship, Small Molecule Libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
