Assessment of predictions in the model quality assessment category
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
The article presents our evaluation of the predictions submitted to the model quality assessment (QA) category in CASP7. In this newly introduced category, predictors were asked to provide quality estimates for protein structure models. The QA category uses the automatically produced models that are traditionally distributed to CASP participants as input for predictions. Predictors were asked to provide an index of the quality of these individual models (QM1) as well as an index for the expected correctness of each of their residues (QM2). We computed the correlation between the observed and predicted quality of the models and of the individual residues achieved by the participating groups and evaluated the statistical significance of the differences. We also compared the results with those obtained by a "naïve predictor" that assigns a quality score related to how close the model is to the structure of the most similar protein of known structure. The aims of a method for assessing the overall quality of a model can be twofold: selecting the best (or one of the best) model(s) among a set of plausible choices, or assigning a nonrelative quality value to an individual model. The applications of the two strategies are different, albeit equally important. Our assessment of the QA category demonstrates that methods for addressing the first task effectively do exist, while there is room for improvement as far as the second aspect is concerned. Notwithstanding the limited number of groups submitting predictions for residue-level accuracy, our data demonstrate that a respectable accuracy in this task can be achieved by methods relying on the comparison of different models for the same target.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
The relation between the divergence of sequence and structure in proteins.
Chothia, C., & Lesk, A. M. (1986) · The EMBO Journal · doi:10.1002/j.1460-2075.1986.tb04288.x · Cited by 2,458
Protein Structure Prediction and Structural Genomics
Baker, D., & Sali, A. (2001) · Science · doi:10.1126/science.1065659 · Cited by 1,587
LGA: a method for finding 3D similarities in protein structures
Zemla, A. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg571 · Cited by 1,069
Database resources of the National Center for Biotechnology Information
Wheeler, D. L., Barrett, T., Benson, D. A., Bryant, S. H., Canese, K., Chetvernin, V., Church, D. M., DiCuccio, M., Edgar, R., Federhen, S., et al. (2007) · Nucleic Acids Research · doi:10.1093/nar/gkl1031 · Cited by 734
Cited by
I-TASSER server for protein 3D structure prediction
Zhang, Y. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-40 · Cited by 5,124
Guex, N., Peitsch, M. C., & Schwede, T. (2009) · ELECTROPHORESIS · doi:10.1002/elps.200900140 · Cited by 2,097
Macromolecular Modeling with Rosetta
Das, R., & Baker, D. (2008) · Annual Review of Biochemistry · doi:10.1146/annurev.biochem.77.062906.171838 · Cited by 989
QMEAN server for protein model quality estimation
Benkert, P., Künzli, M., & Schwede, T. (2009) · Nucleic Acids Research · doi:10.1093/nar/gkp322 · Cited by 873
Critical assessment of methods of protein structure prediction (CASP)—Round XII
Moult, J., Fidelis, K., Kryshtafovych, A., Schwede, T., & Tramontano, A. (2018) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25415 · Cited by 326
Self-citationCritical assessment of methods of protein structure prediction—Round VII
Moult, J., Fidelis, K., Kryshtafovych, A., Rost, B., Hubbard, T., & Tramontano, A. (2007) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.21767 · Cited by 324
Self-citationPredicting the accuracy of protein–ligand docking on homology models
Bordogna, A., Pandini, A., & Bonati, L. (2011) · Journal of Computational Chemistry · doi:10.1002/jcc.21601 · Cited by 107
New prediction categories in CASP15
Kryshtafovych, A., Antczak, M., Szachniuk, M., Zok, T., Kretsch, R. C., Rangan, R., Pham, P., Das, R., Robin, X., Studer, G., et al. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26515 · Cited by 54
“32 Given the recent advances in the accuracy of protein modeling methods, 11,12 CASP organizers decided to include prediction of protein-and RNA-small molecule ligand complexes into the scope of CASP15 experiment hoping to boost development of methods in this area.”
MethodSelf-citationChallenges in bridging the gap between protein structure prediction and functional interpretation
Varadi, M., Tsenkov, M., & Velankar, S. (2025) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26614 · Cited by 36
Assessing scoring metrics for AlphaFold2 and AlphaFold3 protein complex predictions
Genz, L. R., Nair, S., Nagar, N., & Topf, M. (2025) · Protein Science · doi:10.1002/pro.70327 · Cited by 17
All‐Atom Protein Sequence Design Based on Geometric Deep Learning
Liu, J., Guo, Z., You, H., Zhang, C., & Lai, L. (2024) · Angewandte Chemie International Edition · doi:10.1002/anie.202411461 · Cited by 10
High-accuracy protein model quality assessment using attention graph neural networks
Zhang, P., Xia, C., & Shen, H.-B. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbac614 · Cited by 9
New prediction categories in CASP15
Kryshtafovych, A., Antczak, M., Szachniuk, M., Zok, T., Kretsch, R. C., Rangan, R., Pham, P., Das, R., Robin, X., Studer, G., et al. (2023) · doi:10.22541/au.168371643.39402261/v1 · Cited by 4
Self-citation
Metadata
- Authors
- Cozzetto, D., Kryshtafovych, A., Ceriani, M., & Tramontano, A.
- Venue
- Proteins: Structure, Function, and Bioinformatics
- Publisher
- Wiley
- Published
- 2007
- Volume
- 69
- Issue
- S8
- Pages
- 175-183
- ISSN
- 0887-3585 (ISSN-L)
- 1097-0134
- OpenAlex
- W2048290645
- Cited by
- 122
- References
- 5
- FWCI
- 6.35
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.