The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Real External Predictivity of QSAR Models. Part 2. New Intercomparable Thresholds for Different Validation Criteria and the Need for Scatter Plot Inspection
Open accessAbstract
The evaluation of regression QSAR model performance, in fitting, robustness, and external prediction, is of pivotal importance. Over the past decade, different external validation parameters have been proposed: Q(F1)(2), Q(F2)(2), Q(F3)(2), r(m)(2), and the Golbraikh-Tropsha method. Recently, the concordance correlation coefficient (CCC, Lin), which simply verifies how small the differences are between experimental data and external data set predictions, independently of their range, was proposed by our group as an external validation parameter for use in QSAR studies. In our preliminary work, we demonstrated with thousands of simulated models that CCC is in good agreement with the compared validation criteria (except r(m)(2)) using the cutoff values normally applied for the acceptance of QSAR models as externally predictive. In this new work, we have studied and compared the general trends of the various criteria relative to different possible biases (scale and location shifts) in external data distributions, using a wide range of different simulated scenarios. This study, further supported by visual inspection of experimental vs predicted data scatter plots, has highlighted problems related to some criteria. Indeed, if based on the cutoff suggested by the proponent, r(m)(2) could also accept not predictive models in two of the possible biases (location, location plus scale), while in the case of scale shift bias, it appears to be the most restrictive. Moreover, Q(F1)(2) and Q(F2)(2) showed some problems in one of the possible biases (scale shift). This analysis allowed us to also propose recalibrated, and intercomparable for the same data scatter, new thresholds for each criterion in defining a QSAR model as really externally predictive in a more precautionary approach. An analysis of the results revealed that the scatter plot of experimental vs predicted external data must always be evaluated to support the statistical criteria values: in some cases high statistical parameter values could hide models with unacceptable predictions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Golbraikh, A., & Tropsha, A. (2002) · Journal of Molecular Graphics and Modelling · doi:10.1016/s1093-3263(01)00123-1 · Cited by 3,735
Principles of QSAR models validation: internal and external
Gramatica, P. (2007) · QSAR & Combinatorial Science · doi:10.1002/qsar.200610151 · Cited by 2,208
Self-citationTropsha, A., Gramatica, P., & Gombar, V. K. (2003) · QSAR & Combinatorial Science · doi:10.1002/qsar.200390007 · Cited by 2,188
Self-citationBest Practices for QSAR Model Development, Validation, and Exploitation
Tropsha, A. (2010) · Molecular Informatics · doi:10.1002/minf.201000061 · Cited by 2,024
Chirico, N., & Gramatica, P. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci200211n · Cited by 779
Self-citationComments on the Definition of the Q2 Parameter for QSAR Validation
Consonni, V., Ballabio, D., & Todeschini, R. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci900115y · Cited by 585
Further exploring r metrics for validation of QSPR models
Ojha, P. K., Mitra, I., Das, R. N., & Roy, K. (2011) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2011.03.011 · Cited by 554
Schüürmann, G., Ebert, R.-U., Chen, J., Wang, B., & Kühne, R. (2008) · Journal of Chemical Information and Modeling · doi:10.1021/ci800253u · Cited by 531
Comparative Studies on Some Metrics for External Validation of QSPR Models
Roy, K., Mitra, I., Kar, S., Ojha, P. K., Das, R. N., & Kabir, H. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci200520g · Cited by 460
Cited by
Bouarab-Chibane, L., Forquet, V., Lantéri, P., Clément, Y., Léonard-Akkari, L., Oulahal, N., Degraeve, P., & Bordes, C. (2019) · Frontiers in Microbiology · doi:10.3389/fmicb.2019.00829 · Cited by 744
Gramatica, P., & Sangion, A. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00088 · Cited by 420
Self-citationPrinciples of QSAR Modeling: Comments and Suggestions From Personal Experience
Gramatica, P. (2020) · International Journal of Quantitative Structure-Property Relationships · doi:10.4018/ijqspr.20200701.oa1 · Cited by 258
Self-citation
Metadata
- Authors
- Chirico, N., & Gramatica, P.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2012-07-13
- Volume
- 52
- Issue
- 8
- Pages
- 2044-2058
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2019657322
- Cited by
- 536
- References
- 20
- FWCI
- 28.85
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.