Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Scoring model structure is an essential component of protein structure prediction that can affect the prediction accuracy tremendously. Users of protein structure prediction results also need to score models to select the best models for their application studies. In Critical Assessment of techniques for protein Structure Prediction (CASP), model accuracy estimation methods have been tested in a blind fashion by providing models submitted by the tertiary structure prediction servers for scoring. In CASP13, model accuracy estimation results were evaluated in terms of both global and local structure accuracy. Global structure accuracy estimation was evaluated by the quality of the models selected by the global structure scores and by the absolute estimates of the global scores. Residue‐wise, local structure accuracy estimations were evaluated by three different measures. A new measure introduced in CASP13 evaluates the ability to predict inaccurately modeled regions that may be improved by refinement. An intensive comparative analysis on CASP13 and the previous CASPs revealed that the tertiary structure models generated by the CASP13 servers show very distinct features. Higher consensus toward models of higher global accuracy appeared even for free modeling targets, and many models of high global accuracy were not well optimized at the atomic level. This is related to the new technology in CASP13, deep learning for tertiary contact prediction. The tertiary model structures generated by deep learning pose a new challenge for EMA (estimation of model accuracy) method developers. Model accuracy estimation itself is also an area where deep learning can potentially have an impact, although current EMA methods have not fully explored that direction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mariani, V., Biasini, M., Barbato, A., & Schwede, T. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt473 · Cited by 1,138
LGA: a method for finding 3D similarities in protein structures
Zemla, A. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg571 · Cited by 1,069 · cited 1×
“6 Two model accuracy measures, GDT-TS 9 for global fold accuracy and LDDT 10 for local environment accuracy, were adopted after testing several available measures.”
MethodAccurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
Wang, S., Sun, S., Li, Z., Zhang, R., & Xu, J. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005324 · Cited by 931
Protein structure determination using metagenome sequence data
Ovchinnikov, S., Park, H., Varghese, N., Huang, P.-S., Pavlopoulos, G. A., Kim, D. E., Kamisetty, H., Kyrpides, N. C., & Baker, D. (2017) · Science · doi:10.1126/science.aah4043 · Cited by 520
Cited by
Critical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506
Self-citationCritical assessment of methods of protein structure prediction (CASP)—Round XIV
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26237 · Cited by 503
Self-citationImproved protein structure refinement guided by deep learning based accuracy estimation
Hiranuma, N., Park, H., Baek, M., Anishchenko, I., Dauparas, J., & Baker, D. (2021) · Nature Communications · doi:10.1038/s41467-021-21511-x · Cited by 243
Self-citationIns and outs of AlphaFold2 transmembrane protein structure predictions
Hegedűs, T., Geisler, M., Lukács, G. L., & Farkas, B. (2022) · Cellular and Molecular Life Sciences · doi:10.1007/s00018-021-04112-1 · Cited by 127
New prediction categories in CASP15
Kryshtafovych, A., Antczak, M., Szachniuk, M., Zok, T., Kretsch, R. C., Rangan, R., Pham, P., Das, R., Robin, X., Studer, G., et al. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26515 · Cited by 54
Self-citation
Metadata
- Authors
- Won, J., Baek, M., Monastyrskyy, B., Kryshtafovych, A., & Seok, C.
- Venue
- Proteins: Structure, Function, and Bioinformatics
- Publisher
- Wiley
- Published
- 2019-08-30
- Volume
- 87
- Issue
- 12
- Pages
- 1351-1360
- ISSN
- 0887-3585 (ISSN-L)
- 1097-0134
- OpenAlex
- W2969735716
- Cited by
- 61
- References
- 4
- FWCI
- 3.22
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning. https://app.cochat.ai/papers/10.1002/prot.25804
- The paper
- Won, J., Baek, M., Monastyrskyy, B., Kryshtafovych, A., & Seok, C. (2019). Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning. Proteins: Structure, Function, and Bioinformatics. https://doi.org/10.1002/prot.25804
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.