CoChat
Sign Up
Cited by 503Protein Structure and DynamicsEnzyme Structure and FunctionMachine Learning in Bioinformatics

Critical assessment of methods of protein structure prediction (CASP)—Round XIV

Open access
Kryshtafovych, Andriy, Schwede, Torsten, Topf, Maya, Fidelis, Krzysztof, Moult, John·2021·Proteins: Structure, Function, and Bioinformatics
DOI 10.1002/prot.26237PMC PMC8726744
Open this paper in CoChat — free accountOpen PDF

Abstract

Critical assessment of structure prediction (CASP) is a community experiment to advance methods of computing three‐dimensional protein structure from amino acid sequence. Core components are rigorous blind testing of methods and evaluation of the results by independent assessors. In the most recent experiment (CASP14), deep‐learning methods from one research group consistently delivered computed structures rivaling the corresponding experimental ones in accuracy. In this sense, the results represent a solution to the classical protein‐folding problem, at least for single proteins. The models have already been shown to be capable of providing solutions for problematic crystal structures, and there are broad implications for the rest of structural biology. Other research groups also substantially improved performance. Here, we describe these results and outline some of the many implications. Other related areas of CASP, including modeling of protein complexes, structure refinement, estimation of model accuracy, and prediction of inter‐residue contacts and distances, are also described.

AI snapshot

Written from the paper’s full text.

TL;DR

CASP14 found that deep learning methods, especially AlphaFold2 (group 427), produced protein structure models competitive with experimental accuracy for at least two-thirds of targets, representing a solution to the classical single-protein folding problem. Other groups and servers also improved substantially, while CASP’s other categories showed less dramatic progress.

Key findings

  • AlphaFold2 models were competitive with experimental accuracy for at least 2/3 of targets; the CASP14 best-model trend line started at about GDT_TS 95 and finished at about 85 for difficult targets, with about 2/3 of the 96 targets exceeding the roughly 90 GDT_TS experimental limit. Only four targets had another group obtain a higher GDT_TS.
  • Other groups advanced: CASP14 server performance was similar to the best all-group performance in CASP13, even though AlphaFold2 did not submit server models.
  • In refinement, only four methods on average improved models over starting structures, and no method improved much more than half of targets; no refinement approached AlphaFold2 accuracy. Contact accuracy showed no further improvement, while the first assessment of inter-residue distance probability distributions provided a baseline.
  • For estimation of model accuracy, effective methods had an average loss of about 10 GDT_TS units; AlphaFold2’s self-estimated Cα error had an average normalized accuracy (ASE) of 0.91 out of 1.0. In protein complexes, 39 groups modeled 22 targets, but overall progress was small.

Methods

  • CASP14 was a blind community experiment: 97 research groups from 19 countries tested 215 modeling methods and submitted over 67,000 predictions in six categories.
  • Experimental structures were received for 52 proteins and protein-protein complexes: 42 by X-ray crystallography, seven by cryo-EM, and three by NMR. These were released as 68 tertiary structure modeling targets and reorganized into 96 evaluation units; 22 quaternary structure targets were divided into 29 evaluation units, with 12 targets also in the joint CASP/CAPRI experiment.
  • Independent assessors evaluated submitted models using metrics including GDT_TS and Cα-RMSD, with targets classified as TBM-Easy, TBM-Hard, FM/TBM, or FM. Assessed categories included single-protein structure, complexes, refinement, estimation of model accuracy, inter-residue contacts and distances, function, and domain interactions.

Limitations

  • The paper states that data are limited and some contributing factors are correlated, complicating interpretation; cause and effect are hard to separate conclusively between target difficulty, experimental data quality, and model performance.
  • The paper states that CASP cannot insist on full code and data availability, and notes controversy over the delay in AlphaFold2 methodology and code release.
  • The paper states that a more real-life SARS-CoV-2 modeling test suggested the CASP evaluation framework for model accuracy estimation may be misleading: there was little agreement on the most accurate structures, and only one EMA method selected an AlphaFold model as best, for only one target. The text also makes evident that Covid-19 prevented data-assisted modeling and forced a virtual conference.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1002/prot.26237

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 39 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Other research groups also substantially improved performance in CASP14.

    “Other research groups also substantially improved performance.”
    • Also stated in: § Three-dimensional Protein Structure Modeling · plain
      “Other groups also showed substantial improvement.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    In CASP14, deep learning methods from one research group (AlphaFold2) consistently delivered computed protein structures rivalling the corresponding experimental structures in accuracy.

    “In the most recent experiment (CASP14) deep learning methods from one research group consistently delivered computed structures rivalling the corresponding experimental ones in accuracy.”
    Quote in the paperSubstantial claim
  • Findingown data

    In CASP14, AlphaFold2 (group 427) submitted models competitive with experimental accuracy for at least 2/3 of the targets.

    “One research group, AlphaFold2 from the company DeepMind, submitted models competitive with experimental accuracy for at least 2/3 of the targets (group 427 in the Results tables, available online at https://predictioncenter.org/casp14/results.cgi).”
    Quote in the paperSubstantial claim
  • Findingown data

    CASP14 saw an extraordinary increase in the accuracy of the computed three-dimensional protein structures.

    “CASP14 saw an extraordinary increase in the accuracy of the computed three-dimensional protein structures.”
    Quote in the paperSubstantial claim
  • Findingown data

    Performance of servers in CASP14 is similar to the best performance of all groups in CASP13.

    “Also of note, performance of servers in CASP14 (dotted black line in Figure 1) is similar to the best performance of all groups in CASP13.”
    Quote in the paperSubstantial claim
  • May combine two claims. 40
  • Classified as probably not a substantial claim. 13
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    The CASP14 trend curve starts at a GDT_TS of about 95, and finishes at about 85 for difficult targets.

    “Astonishingly, the trend curve for CASP14 (the black straight-line) starts at a GDT_TS of about 95, and finishes at about 85 for difficult targets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AlphaFold2 models are generally much more accurate than other models and are the only ones to consistently approach experimental quality.

    “Nevertheless, it is clear that the AlphaFold2 models are generally much more accurate, and the only ones to consistently approach experimental quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Three other targets were classified as 'intertwined' and two as coiled-coils, and these would also be difficult to predict by starting with monomeric structures.

    “Three other targets were classified as ‘intertwined’ and two as coiled-coils, and these would also be difficult to predict by starting with monomeric structures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    A number of groups augmented classical docking with interface residue-residue contact prediction, often using deep-learning methods, producing some of the best results, although gains are fairly modest.

    “A number of groups have now augmented these with the prediction in interface residue-residue contacts, often employing deep-learning methods, for example22, producing some of the best results, although so far, the gains are fairly modest.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Baek M, Anishchenko I, Park H, Humphreys IR, Baker D. Protein oligomer modeling guided by predicted interchain contacts in CASP14. Proteins 2021. doi: 10.1002/prot.26197. Online ahead of print. PMC8616806 34324224 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    About a third of the protein complex targets were determined by cryo-EM.

    “About a third of the targets were determined by cryo-EM, and the increasing output from that technology is allowing larger and more complicated protein assemblies to be included in CASP.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Most methods start with models of individual constituent proteins, and conformational changes accompanying complex formation as well as intertwining of monomers present major challenges.

    “Most methods start with models of individual constituent proteins, and conformational changes accompanying complex formation as well as intertwining of monomers present major challenges.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Of the 105 residues involved, 64 were found to be close to lattice contacts, suggesting the local conformations are determined by the crystal environment.

    “Of the 105 residues involved, he found 64 to be close to lattice contacts, suggesting the local conformations are determined by the crystal environment (also not provided to the participants).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    These are cases where the best models differed from experiment for small regions of the polypeptide chain, and refinement methods were unable to converge to the experimental structure.

    “These are cases where the best models differed from experiment for small regions of the polypeptide chain, and refinement methods were unable to converge to the experimental structure.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the most difficult targets (FM), performance is slightly lower overall than for the easiest category, but about 30% of targets achieve a GDT_TS above 90, and 75% have a GDT_TS of 80 or greater.

    “For the most difficult targets (‘FM’, black line), where no assistance from homologous structures is available, performance is slightly lower overall than for the easiest category, but still about 30% of targets achieve a GDT_TS above 90, and 75% have a GDT_TS of 80 or greater.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    In CASP14, only four methods on average improve models over the starting structures provided, and no method improves much more than half of the targets.

    “In CASP14, only four methods on average improve models over the starting structures provided, and no method improves much more than half of the targets6.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] Simpkin AJ, Sanchez Rodriguez F, Mesdaghi S, Kryshtafovych A, Rigden DJ. Evaluation of model refinement in CASP14. Proteins 2021. doi: 10.1002/prot.26185. Online ahead of print. doi: 10.1002/prot.26209. Online ahead of print. PMC8616799 34288138 source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Although earlier CASP rounds have seen substantial progress, in the last two, that has been harder to identify.

    “Further, although earlier CASP rounds have seen substantial progress, in the last two, that has been harder to identify.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Performance is strongest for TBM-easy targets, with the lowest GDT_TS of 90, and many targets with greater than 95.

    “Performance is still strongest for the category of targets with most information from homologous structures available (‘TBM-easy’, green), with the lowest GDT_TS of 90, and many targets with greater than 95.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There were three NMR targets in CASP14, two of which are template free (FM) targets with very low GDT_TS values.

    “There were also three NMR targets in CASP14 (data not included in the figure) two of which are template free (FM) targets with very low GDT_TS values (blue points in Figure 1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although outstanding performance is dominated by AlphaFold2, other groups also advanced substantially from CASP13.

    “Although this outstanding performance is dominated by AlphaFold2 (group 427), the dashed black line in Figure 1 shows that other groups also advanced substantially from CASP13.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The refinement category assessor, Daniel Rigden, looked at this for a subset of seven targets, with GDT-TS values ranging from 72 to 936.

    “The refinement category assessor, Daniel Rigden, has looked at this for a subset of seven targets, with GDT-TS values ranging from 72 to 936.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    39 groups took part in the protein complex modeling category, with 22 targets altogether, 12 of which were also considered by CAPRI participants.

    “39 groups took part in this modeling category, with 22 targets altogether, 12 of which were also considered by CAPRI participants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    In CASP14, about 2/3 of the 96 targets reached GDT_TS values greater than the expected limit, and so are considered competitive with experiment in backbone accuracy.

    “In CASP14, about 2/3 of the 96 targets reached GDT_TS values greater than that, and so are considered competitive with experiment in backbone accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    This target has very extensive inter-subunit interactions, including a domain swap in which part of the fold of one monomer occupies the corresponding position of a neighboring one.

    “This is an unusual target in that there are very extensive inter-subunit interactions, including a domain swap21 in which part of the fold of one monomer occupies the corresponding position of a neighboring one.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [21] Liu Y, Eisenberg D. 3D domain swapping: as domains continue to swap. Protein Sci 2002; 11(6): 1285– 1299. 12021428 PMC2373619 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Findingown data

    Figure 4A shows a relationship between average best GDT_TS and the quality of experimental data.

    “Figure 4A shows the relationship between average best GDT_TS and the quality of experimental data (three ranges of X-ray structure resolution and cryo-EM).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The threshold of 90 GDT_TS corresponds to approximately 1.5 Angstroms RMSD, and 80 corresponds to about 2.5 Angstroms (all residues included, calculated with LGA 24).

    “The threshold of 90 GDT_TS corresponds to approximately 1.5 Angstroms RMSD, and 80 corresponds to about 2.5 Angstroms (all residues included, calculated with LGA 24).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Zemla A LGA: A method for finding 3D similarities in protein structures. Nucleic Acids Res 2003; 31(13): 3370– 3374. 12824330 PMC168977 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    The startling results for 3D structure tend to overshadow the other areas of CASP in this round, where progress was less dramatic.

    “The startling results for 3D structure tend to overshadow the other areas of CASP in this round, where progress was less dramatic.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    AlphaFold2's network directly outputs confidence estimates for the position of each residue in the structure, and these estimates are impressively accurate.

    “(d) The network directly outputs confidence estimates for the position of each residue in the structure, and as noted earlier, these are impressively accurate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    In the shallow-alignment region, the accuracy spread is large, there are still highly accurate models, some with only a single sequence available, and very few models are really low accuracy.

    “However, the accuracy spread in that region is large, and there are still highly accurate models, some with only a single sequence available.”
    • Also stated in: § DISCUSSION · plain
      “Very few models are really low accuracy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38] Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, Tunyasuvunakool K, Bates R, Zidek A, Potapenko A, Bridgland A, Meyer C, Kohl SAA, Ballard AJ, Cowie A, Romera-Paredes B, Nikolov S, Jain R, Adler J, Back T, Petersen S, Reiman D, Clancy E, Zielinski M, Steinegger M, Pacholska M, Berghammer T, Bodenstein S, Silver D, Vinyals O, Senior AW, Kavukcuoglu K, Kohli P, Hassabis D. Highly accurate protein structure prediction with AlphaFold. Nature 2021; 596(7873): 583– 589. 34265844 PMC8371605 source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Particularly for global estimates, the methods appear to be usefully accurate for selecting close to the best model available, with an average loss of about 10 GDT_TS units for the most effective methods, both single model and consensus based.

    “On the other hand, particularly for global estimates, the methods appear to be usefully accurate for selecting close to the best model available, with an average loss of about 10 GDT_TS units for the most effective methods, both single model and consensus based.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    In CASP13, there was a similar dependency on alignment depth, though with an overall increase in accuracy.

    “In CASP13, where convolutional neural networks had become the most effective methods, there is a similar dependency, though with an overall increase in accuracy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    By far the most accurate models in the SARS2 experiment are from the AlphaFold group, consistent with the later CASP results.

    “By far the most accurate models are from the AlphaFold group, consistent with the later CASP results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    By CASP14, other research groups had substantially surpassed AlphaFold's performance, and publicly available servers were performing at a level similar to AlphaFold's CASP13 level.

    “In spite of this, as Figure 1 shows, other research groups had substantially surpassed AlphaFold’s performance by CASP14, and most importantly, publicly available servers were also performing at a level similar to AlphaFold’s CASP13 level.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The average normalized accuracy in estimated Cα error (ASE) for AlphaFold2's CASP14 models is 0.91 (out of a maximum possible 1.0), suggesting this approach to error estimation can be very effective.

    “The average normalized accuracy in estimated Cα error (ASE) for their CASP14 models is 0.91 (out a maximum of possible 1.0), suggesting this approach to error estimation can be very effective.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingnone given

    In recent CASPs there has not been substantial progress in methods performance for estimation of model accuracy.

    “In recent CASPs there has not been substantial progress in methods performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The AlphaFold2 methodology consistently produces models competitive in accuracy with the best experimental results, and subatomic-scale differences from experiment are the norm.

    “The AlphaFold2 methodology consistently produces models competitive in accuracy with the best experimental results, and subatomic scale differences from experiment are the norm.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Only one EMA method selected an AlphaFold model as the best, for only one of the targets, and generally the AlphaFold models were not highly ranked.

    “But only one EMA method selected an AlphaFold model as the best, for only one of the targets, and generally the AlphaFold models were not highly ranked.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The Baker group have successfully incorporated deep learning prediction of inter-residue distance errors into their refinement procedures, allowing computational effort to be focused on the parts of the structure most needing it.

    “First, the Baker group have successfully incorporated deep learning prediction of inter-residue distance errors into their refinement procedures22 allowing computational effort to be focused on the parts of the structure most needing it.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Baek M, Anishchenko I, Park H, Humphreys IR, Baker D. Protein oligomer modeling guided by predicted interchain contacts in CASP14. Proteins 2021. doi: 10.1002/prot.26197. Online ahead of print. PMC8616806 34324224 source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    There are two free-modeling targets with zinc binding sites and another target with two bound hemes; the parts of these structures interacting with the ligands are accurately modeled even though the ligands are absent in the calculation.

    “For example, there are two free modeling targets with zinc binding sites and another target with two bound hemes.”
    • Also stated in: § DISCUSSION · plain
      “The parts of these structures interacting with the ligands are accurately modeled, even though the ligands are absent in the calculation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Some CASP targets have local disorder and flexibility, and AlphaFold2 sometimes produces a variety of structures for these regions.

    “Some CASP targets do have local disorder and flexibility and it appears that AlphaFold2 sometimes produces a variety of structures for these regions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In CASP13 (2018), with the introduction of effective deep learning methods, the trend line rose to above 60 on the GDT_TS scale, even for the most difficult targets, a major advance from the previous CASP.

    “In CASP13 (2018)18, with the introduction of effective deep learning methods, the trend line rose to above 60 on the GDT_TS scale, even for the most difficult targets, a major advance from the previous CASP.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Kryshtafovych A, Schwede T, Topf M, Fidelis K, Moult J. Critical assessment of methods of protein structure prediction (CASP)-Round XIII. Proteins 2019; 87(12): 1011– 1020. 31589781 PMC6927249 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    There is only minor benefit from homologous structure information; models are only marginally more accurate when it is available.

    “Apparently, there is only minor benefit from homologous structure information - models are only marginally more accurate when it is available.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Analysis by Gaetano Montelione's group shows that one NMR target, T1027, is a dynamic structure and the best computed structures may correspond to a member of the ensemble.

    “Analysis by Gaetano Montelione’s group19, shows that one of these, T1027, is a dynamic structure and the best computed structures may correspond to a member of the ensemble.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Huang YP. Assessment of Prediction Methods for Protein Structures Determined by NMR in CASP14: Impact of AlphaFold2. Proteins 2021 [ CASP14 special issue]. PMC8616817 34559429 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The difference between the T1047s1 model and experiment is not a computational failure, although it is a poorer representation of the in vivo structure.

    “In this sense, this type of difference to experiment is not a computational failure, although it is of course a poorer representation of the in vivo structure.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Other research groups have now surpassed AlphaFold's leading performance in CASP13 and have made these improved methods available in servers.

    “Thus, other research groups have not only now surpassed AlphaFold’s leading performance in CASP13, they have also made these improved methods available in servers, some of which are publicly accessible.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Data are limited and some contributing factors are correlated, complicating interpretation of agreement with experiment.

    “Data are limited and some contributing factors are correlated, complicating interpretation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    Participants were not provided with specific multimer assembly information, and so all submitted models are based on an isolated monomer environment.

    “Participants were not provided with specific multimer assembly information, and so all submitted models are based on an isolated monomer environment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Baker group has already reported modeling accuracy similar to that of AlphaFold2 using deep learning methods.

    “the Baker group has already reported modeling accuracy similar to that of AlphaFold2 using deep learning methods28.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Baek M, DiMaio F, Anishchenko I, Dauparas J, Ovchinnikov S, Lee GR, Wang J, Cong Q, Kinch LN, Schaeffer RD, Millan C, Park H, Adams C, Glassman CR, DeGiovanni A, Pereira JH, Rodrigues AV, van Dijk AA, Ebrecht AC, Opperman DJ, Sagmeister T, Buhlheller C, Pavkov-Keller T, Rathinaswamy MK, Dalwadi U, Yip CK, Burke JE, Garcia KC, Grishin NV, Adams PD, Read RJ, Baker D. Accurate prediction of protein structures and interactions using a three-track neural network. Science 2021; 373(6557): 871– 876. 34282049 PMC7612213 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Refinement may in fact be improving but most of what the methods can deliver is already being exploited in developing the pre-refinement models.

    “That is, refinement may in fact be improving but most of what the methods can deliver is already being exploited in developing the pre-refinement models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This failure may reflect unusual properties of the AlphaFold models.

    “This failure may reflect unusual properties of the AlphaFold models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Given adequate ligand docking methods, allosteric conformational changes should be addressable.

    “Given adequate ligand docking methods (see below), allosteric conformational changes should be addressable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Deep learning methods for small ligand docking have been developing in parallel to the protein structure work.

    “Deep learning methods for small ligand docking have been developing in parallel to the protein structure work51”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [51] Brown N, Ertl P, Lewis R, Luksch T, Reker D, Schneider N. Artificial intelligence in chemistry and drug design. J Comput Aided Mol Des 2020; 34(7): 709– 715. 32468207

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    There is reason to believe that the new methods will also be extended to protein complexes.

    “As discussed earlier, there is reason to believe that the new methods will also be extended to protein complexes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    About only one in 15 prolines in protein structures adopt the cis form.

    “a very finely balanced energy difference with about only one in 15 prolines in protein structures adopting the cis form50.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] MacArthur MW, Thornton JM. Influence of proline residues on protein conformation. J Mol Biol 1991; 218(2): 397– 412. 2010917 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The methods will create more general synergy between computation and experiment.

    “As well as providing a powerful aid for solving structures the methods will create more general synergy between computation and experiment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Short time scale classical dynamics do not appear to be within the scope of the methods, so there is still a role for molecular dynamics.

    “Short time scale classical dynamics do not appear to be within the scope of the methods, so there is still a role there for molecular dynamics.”
    Quote in the paperNot a substantial claim
50 of 50

  • The paper relies on this without support.

    The fold of the protein backbone is usually correct at values above 50 on the GDT_TS scale.

    “Note that the fold of the protein backbone is usually correct at values above 50 on this scale, and so that represented a solution to the problem as classically defined, for most targets.”

  • The paper relies on this without support.

    Because of experimental errors and artifacts, a GDT_TS of 100 is highly unlikely.

    “Because of experimental errors and artifacts, a GDT_TS of 100 is highly unlikely, and previous CASP trend lines intercept the Y axis at about 90, indicating that that is approximately the limit expected.”

  • The paper relies on this without support.

    In previous CASPs, with rare exceptions, differences between models and experiment were usually dominated by computational error.

    “In previous CASPs, with rare exceptions, it was usually safe to assume that differences between models and experiment were dominated by computational error.”

  • The paper relies on this without support.

    Most proteins are not monomers, and most biology involves interactions with other proteins, DNA, RNA, or small molecules.

    “Since the formulation of that problem, science has moved on - most proteins are not monomers, and most biology involves interactions with other proteins, DNA, RNA, or small molecules.”

  • The authors state this as an assumption.

    The CASP refinement category was introduced on the basis that informatics methods are ultimately limited in accuracy and so physics or related representations of atomic interactions together with local conformational exploration would be essential to achieving atomic accuracy.

    “The CASP refinement category was introduced in CASP8 (2008) on the basis that informatics methods are ultimately limited in accuracy and so physics or related representations of atomic interactions together with some form of local conformational exploration (such as those provided by conventional molecular dynamics) would be essential to achieving atomic accuracy.”

  • The paper relies on this without support.

    There are astronomically more atomic configurations than are present in the PDB.

    “at this level of accuracy there are astronomically more atomic configurations than are present in the PDB”

  • The paper relies on this without support.

    The CASP14 target set is representative of single, ordered proteins generally.

    “Objective testing in CASP14 has shown that the problem of computing atomic accuracy protein structures from amino acid sequence is solved, at least for single, ordered, proteins.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The second test set consists of 51 targets from the CASP14 competition.

    “Our second test set consists of 51 targets from the CASP14 competition (37).”

    From Evolutionary-scale prediction of atomic level protein structure with a language model (2022) · cites this paper as [37]

    The Introduction describes CASP14 modeling targets, supporting their use as a source for a test set. It reports 68 tertiary-structure modeling targets, but does not identify the citing authors’ 51-target selection or verify its size. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • By extracting residue-residue covariances from MSAs, these algorithms greatly outperform previous approaches and predict structures with atomic-level accuracy.

    “By extracting residue-residue covariances from these MSAs, these algorithms have been shown to greatly outperform previous approaches, including physics-based models, homology-based methods and convolutional neural networks (16) and predict structures with atomic-level accuracy for the first time in history.”

    From High-resolution <i>de novo</i> structure prediction from primary sequence (2022) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • One DOI is assigned to two different papers

    Minor

    Reference 6 assigns DOI 10.1002/prot.26209 to the refinement paper, while reference 8 assigns the same DOI to a different paper on binding properties.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J.
Affiliations
  • University of California, Davis (US)
  • SIB Swiss Institute of Bioinformatics (CH)
  • Leibniz Institute of Virology (LIV) (DE)
  • Institute for Bioscience and Biotechnology Research (US)
Corresponding author
Moult, John
ORCID
Venue
Proteins: Structure, Function, and Bioinformatics
Publisher
Wiley
Published
2021-10-07
Volume
89
Issue
12
Pages
1607-1617
ISSN
  • 0887-3585 (ISSN-L)
  • 1097-0134
DOI
10.1002/prot.26237
PMC
PMC8726744
OpenAlex
W3198923619
Cited by
503
References
50
Citations by year
2026: 60 · 2025: 96 · 2024: 117 · 2023: 136 · 2022: 83 · 2021: 11
FWCI
27.28
Type
journal article
Language
en
Open access
Licence
Locations
  • Proteins Structure Function and Bioinformatics (publishedVersion)
  • PubMed (publishedVersion)
  • edoc (University of Basel) (acceptedVersion, other-oa)
  • PubMed Central (acceptedVersion)
  • Desy Publications Database (Deutsches Elektronen-Synchrotron DESY) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • edoc (University of Basel)
Funders
National Institute of General Medical Sciences (R01GM100482)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
protein structure prediction, CASP14, blind test, deep learning, protein folding, protein complexes, structure refinement, model accuracy assessment, inter-residue contacts
MeSH terms
Amino Acid Sequence, Protein Conformation, Proteins, Software, Models, Statistical, Protein Folding, Computational Biology, Sequence Analysis, Protein, Molecular Dynamics Simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review