CoChat
Sign Up
PreprintCited by 392Protein Structure and DynamicsMachine Learning in BioinformaticsEnzyme Structure and Function

High-resolution de novo structure prediction from primary sequence

Open access
Wu, Ruidong, Ding, Fan, Wang, Rui, Shen, Rui, Zhang, Xiwen, Luo, Shitong, Su, Chenpeng, Wu, Zuofan +4 more·2022·bioRxiv (Cold Spring Harbor Laboratory)
DOI 10.1101/2022.07.21.500999
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent breakthroughs have used deep learning to exploit evolutionary information in multiple sequence alignments (MSAs) to accurately predict protein structures. However, MSAs of homologous proteins are not always available, such as with orphan proteins or fast-evolving proteins like antibodies, and a protein typically folds in a natural setting from its primary amino acid sequence into its three-dimensional structure, suggesting that evolutionary information and MSAs should not be necessary to predict a protein’s folded form. Here, we introduce OmegaFold, the first computational method to successfully predict high-resolution protein structure from a single primary sequence alone. Using a new combination of a protein language model that allows us to make predictions from single sequences and a geometry-inspired transformer model trained on protein structures, OmegaFold outperforms RoseTTAFold and achieves similar prediction accuracy to AlphaFold2 on recently released structures. OmegaFold enables accurate predictions on orphan proteins that do not belong to any functionally characterized protein family and antibodies that tend to have noisy MSAs due to fast evolution. Our study fills a much-encountered gap in structure prediction and brings us a step closer to understanding protein folding in nature.

AI snapshot

Written from the paper’s full text.

TL;DR

OmegaFold predicts high-resolution protein structure from a single primary sequence without MSAs or templates by combining a pretrained protein language model (OmegaPLM) with a geometry-inspired transformer (Geoformer). It achieves comparable accuracy to AlphaFold2 and outperforms RoseTTAFold on recent CAMEO/CASP targets, and improves predictions for antibody CDR-H3 loops and orphan proteins.

Key findings

  • On 146 recent CAMEO single-chain proteins, OmegaFold reached a mean LDDT of 0.82, compared with RoseTTAFold at 0.75 and AlphaFold2 at 0.86 when those methods used MSAs. On 29 challenging CASP free-modeling targets, OmegaFold averaged TM-score 0.79, RoseTTAFold 0.81, and AlphaFold2 0.79.
  • With only single-sequence input, AlphaFold2 and RoseTTAFold were “statistically highly inferior” to OmegaFold; OmegaFold requires only a single amino-acid sequence and does not rely on MSAs or known structures as templates.
  • On 33 recently released nanobody/antibody proteins, OmegaFold predicted CDR-H3 loops with RMSD 2.12 Å versus AlphaFold2 2.98 Å (P=0.0017). On 19 recently released orphan proteins with no homologous sequences identified, OmegaFold reached TM-score 0.73 versus AlphaFold2 0.60 (P=0.0238).
  • Ablation validation LDDT: OmegaFold ensemble 0.816, single OmegaFold 0.801, AlphaFold2 with single-sequence input 0.344, retrained AlphaFold2 with single-sequence input 0.476, and OmegaPLM with structure module but no Geoformer 0.198. Contact-map refinement improved TopL accuracy from 0.393 to 0.704 and reduced geometric inconsistency from 0.638 to 0.104.

Methods

  • OmegaFold combines OmegaPLM, a transformer-based protein language model pretrained on unaligned, unlabeled sequences from UniRef50 (2021/04), with Geoformer, a geometry-inspired transformer using 50 Geoformer layers and an 8-layer structure module.
  • The full model was jointly trained on ~110,000 single-chain structures from the PDB deposited before 2021 and all single domains from SCOP v1.75 with at most 40% sequence identity cutoff; later-released structures were used for validation and hyperparameter selection.
  • Evaluation used a CASP set of 29 challenging free-modeling targets from CASP13/CASP14 and a CAMEO set of 146 recent single-chain proteins from the first six months of 2022, with metrics including LDDT, TM-score, RMSD, pLDDT, contact accuracy, geometric inconsistency, ablation, and runtime comparisons.

Limitations

  • The paper does not include a dedicated limitations section.
  • The text states that long proteins were clipped to the first 1,024 residues in validation and that training cropped sequences to 256, 320, or 384 residues to fit GPU memory, which may limit coverage of long proteins.
  • Evident from the text: antibody and orphan-protein benchmarks are small (33 and 19 proteins), comparisons to AlphaFold2/RoseTTAFold use their default MSA mode, and no prospective experimental validation beyond PDB/CASP/CAMEO benchmarks is described.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2022.07.21.500999

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 15 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Contacts and distances predicted by OmegaPLM alone are reasonably accurate.

    “We found that the contacts and distances predicted by OmegaPLM alone are reasonably accurate.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    OmegaFold's overall runtime is much faster than AlphaFold2 and ColabFold-AF2.

    “Since MSAs are no longer required for OmegaFold to achieve high-resolution performance, its overall runtime is much faster than AlphaFold2, as well as the latest highly-optimized ColabFold-AF2 (40) (Fig1.C).”
    Quote in the paperSubstantial claim
  • Findingown data

    OmegaFold outperforms RoseTTAFold on recently released structures.

    “OmegaFold outperforms RoseTTAFold and achieves similar prediction accuracy to AlphaFold2 on recently released structures.”
    Quote in the paperSubstantial claim
  • Findingown data

    Ablation studies on component parts of OmegaFold show that the components contribute to the performance of the overall model.

    “We also demonstrated through ablation studies on component parts of OmegaFold, that the components contribute to the performance of the overall model (Table S3).”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    OmegaFold enables accurate predictions on orphan proteins that do not belong to any functionally characterized protein family.

    “OmegaFold enables accurate predictions on orphan proteins that do not belong to any functionally characterized protein family and antibodies that tend to have noisy MSAs due to fast evolution.”
    Quote in the paperSubstantial claim
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 14
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingreasoning

    AlphaFold2 is unreliable and has very low predicted LDDT (pLDDT) scores on antibody CDRs.

    “As a result, methods like AlphaFold2 are unreliable and have very low predicted LDDT (pLDDT) scores (Fig. 2A).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    IEEE float16 causes overflow issues during training and IEEE float32 slows computation considerably.

    “We do not change this setting as IEEE float16 causes overflow issues during training and IEEE float32 slows computation considerably.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    OmegaFold does not rely on MSAs nor known structures as templates.

    “Notably, OmegaFold requires only a single amino-acid sequence for protein structure prediction, does not rely on MSAs nor known structures as templates, and scales roughly ten-times faster with comparable or better accuracy to MSA-based methods such as AlphaFold2 and RoseTTAFold.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Recycling improves prediction accuracy with the number of recycles and saturates after about 10 recycles.

    “We find that with the prediction accuracy improves with the number of recycles and saturates after about 10 recycles (Fig. S2).”
    • Also stated in: § 2.9 Ablation Study · plain
      “OmegaFold reaches reasonable performance without recycling, and achieves highest performance after around 10 recycles.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On the CAMEO dataset, OmegaFold structures had a mean LDDT score of 0.82, comparable to RoseTTAFold structures (0.75 mean LDDT score) and AlphaFold2 structures (0.86 mean LDDT score) predicted from MSAs.

    “On the CAMEO dataset, OmegaFold structures had a mean local-distance difference test (LDDT) score of 0.82, with comparable accuracy to RoseTTAFold structures (0.75 mean LDDT score) and AlphaFold2 structures (0.86 mean LDDT score) predicted from MSAs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    OmegaPLM is able to capture structural and functional information encoded in the amino-acid sequences through the embeddings.

    “OmegaPLM is able to capture structural and functional information encoded in the amino-acid sequences through the embeddings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    OmegaPLM consumes less resources and performs better than other protein language models.

    “OmegaPLM strikes a good balance between efficiency and performance, as shown in Figure S1 and Table S1, where OmegaPLM consumes less resources and performs better.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both OmegaPLM and Geoformer have a critical contribution to OmegaFold's superior performance.

    “Table S4 indicates that both OmegaPLM and Geoformer have a critical contribution to OmegaFold's superior performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    OmegaFold scales roughly ten-times faster with comparable or better accuracy to MSA-based methods such as AlphaFold2 and RoseTTAFold.

    “Notably, OmegaFold requires only a single amino-acid sequence for protein structure prediction, does not rely on MSAs nor known structures as templates, and scales roughly ten-times faster with comparable or better accuracy to MSA-based methods such as AlphaFold2 and RoseTTAFold.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    ProtT5 cannot fit on our testing GPU (Nvidia A100 80GB) with the same data size during training.

    “ProtT5 cannot fit on our testing GPU (Nvidia A100 80GB) with the same data size during training.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    OmegaFold performs significantly better than AlphaFold2 in modeling antibody CDR H3 regions and predicting structures of orphan proteins.

    “OmegaFold performs significantly better than AlphaFold2 in modeling antibody CDR H3 regions and predicting structures of orphan proteins.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    OmegaPLM strikes a great balance between efficiency and performance.

    “OmegaPLM strikes a great balance between efficiency and performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On 19 recently released orphan proteins with no homologous sequences identified, OmegaFold predictions (TM-score=0.73) are better than AlphaFold2 predictions (TM-score=0.60), with a P-value of 0.0238.

    “The scatter plot shows comparisons on 19 recently released orphan proteins with no homologous sequences identified.Overall, OmegaFold predictions (TM-score=0.73)are better than AlphaFold2 predictions (TM-score=0.60),with a P-value of 0.0238.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On 33 recently released nanobody and antibody proteins with high-resolution experimental structures, OmegaFold predictions (RMSD=2.12Å) are significantly better than AlphaFold2 predictions (RMSD=2.98Å), with a P-value of 0.0017.

    “The scatter plot depicts the comparison on 33 recently released nanobody and antibody proteins with highresolution experimental structures.Overall, OmegaFold predictions (RMSD=2.12Å)are significantly better than AlphaFold2 predictions (RMSD=2.98Å), with a P-value of 0.0017.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the CASP dataset, OmegaFold structures had an average TM-score of 0.79, slightly lower than RoseTTAFold structures (0.81 mean TM-score) and equivalent to AlphaFold2 structures (0.79 mean TM-score).

    “On the more challenging CASP dataset, OmegaFold structures were also quite accurate with an average TM-score-a common metric for assessing the topological similarity of protein structures-of 0.79, slightly lower than that of RoseTTAFold structures (0.81 mean TM-score) and equivalent to AlphaFold2 structures (0.79 mean TM-score).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    A protein typically folds in a natural setting from its primary amino acid sequence into its three-dimensional structure.

    “Half a century after Anfinsen demonstrated a connection between a protein's amino acid sequence and folded three-dimensional conformation, scientists have built deep learning models that finally can predict high-resolution protein structures.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    EdgeAttention designs one universal function to implement the four functions of AlphaFold2, significantly reducing the computational burden.

    “In this way, we design one universal function to implement the four functions of AlphaFold2, which significantly reduces the computational burden.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: AlphaFold2's 3D structure prediction is driven by triangle attention on edge representations and Invariant Point Attention in the Structure Module.
    • Rests on an assumption: Triangle attention relies on the triangle multiplicative update on edge representations to enforce the edge representation to satisfy the triangle inequality of amino-acid distances, which significantly improves the feasibility of predicted structures based on MSA data.

  • May combine two claims.

    Interpretationreasoning

    OmegaFold fills the interim gap and predicts from sequences for which MSAs are difficult to construct.

    “As more sequencing data accumulates to feed MSA-based methods, OmegaFold fills the interim gap, as well as importantly predicts from sequences for which MSAs are difficult to construct.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Geoformer is designed to optimize high-dimensional representations to capture complex interaction patterns among amino acids while still maintaining geometric consistency in the Euclidean space.

    “To address this problem, we design a new structural module, named Geoformer, which could optimize high-dimensional representations to capture complex interaction patterns among amino acids while still maintaining geometric consistency in the Euclidean space.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The updated edge representation contains information indirectly encoded from the 3D space.

    “In this way, the updated edge representation contains the information indirectly encoded from the 3D space.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    By extracting residue-residue covariances from MSAs, these algorithms greatly outperform previous approaches and predict structures with atomic-level accuracy.

    “By extracting residue-residue covariances from these MSAs, these algorithms have been shown to greatly outperform previous approaches, including physics-based models, homology-based methods and convolutional neural networks (16) and predict structures with atomic-level accuracy for the first time in history.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [16] 2021, Critical assessment of methods of protein structure prediction (CASP)-Round XIV, Proteins source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The original GAU suggests that relu^2 performs better compared to softmax in terms of both computation speed and convergence rate.

    “The original GAU in (25) also suggests that relu 2 (•) performs better compared to softmax in terms of both computation speed and convergence rate.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] 2021, Learning the protein language: Evolution, structure, and function, Cell Syst source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Proteins having sequence similarity greater than 25% or BLAST E-value less than 0.1 with any test protein were excluded from training data.

    “Proteins having sequence similarity > 25% or BLAST E-value < 0.1 with any test protein were excluded from training data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The deeper layers of current transformer-based PLMs focus relatively more on learning protein contacts, whereas secondary structure information is spread more evenly across both shallow and deep layers.

    “An important observation is that the deeper layers of current transformer-based PLMs focus relatively more on learning protein contacts, whereas the secondary structure information is spread more evenly across both shallow and deep layers (15).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The ease of predicting many amino acids from short-range context creates an easy prediction task for the model to learn and causes the model to overly focus on short-range relations.

    “This creates an easy prediction task for the model to learn and causes the model to overly focus on short-range relations.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous deep learning models usually sample a sub-sequence with fixed length.

    “Previous deep learning models usually sample a sub-sequence with fixed length (20).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] 2022, AlphaFold2: A role for disordered protein/region prediction?, Int. J. Mol. Sci source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Evolutionary information may well be encoded in primary sequences, which can then be used as features for structure prediction.

    “We give further evidence that such evolutionary information may well be encoded in primary sequences, which can then be used as features for structure prediction.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The Gated Attention Unit (GAU) has shown great promise as an alternative to multi-headed self-attention with smaller memory consumption and faster convergence rate.

    “Instead of using multi-headed self-attention (MHSA), we adopt the Gated Attention Unit (GAU) (line 8 in Algorithm 1), which has shown great promise as an alternative to MHSAs with smaller memory consumption and faster convergence rate (25).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25] 2021, Learning the protein language: Evolution, structure, and function, Cell Syst source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Antibody complementarity-determining regions (CDRs) are the most diverse and variable parts of the molecules.

    “Antibody complementarity-determining regions (CDRs) are the most diverse and variable parts of the molecules (38,39).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [38] source

      Not checked in the free run.

    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    OmegaPLM is trained for 3D structure prediction, utilizing 99.99% of the data for training and the rest for validation.

    “Since OmegaPLM is trained for 3D structure prediction, we utilize 99.99% of the data for training and the rest of sequences for validation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Interpretationnone given

    F and G can be calculated by using the recursion formula given in Equations 5 and 6.

    “F and G can be calculated by using the following recursion formula,”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The maximum value of Equation 4 is obtained at G_{L,N_crop}.

    “The maximum value of Equation 4 is obtained at G L,Ncrop .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Equation 4 can be solved by dynamic programming in a very efficient way.

    “Equation 4 could be solved by dynamic programming in a very efficient way.”
    Quote in the paperNot a substantial claim
33 of 33

  • The paper relies on this without support.

    Prediction accuracies for advanced MSA-based methods drop sharply in the absence of a multitude of sequence homologs from which to construct MSAs.

    “However, prediction accuracies for all these advanced methods drop sharply in the absence of a multitude of sequence homologs from which to construct MSAs.”

  • The paper relies on this without support.

    MSAs on antibody CDRs, especially the CDR3 loops on the heavy chain, are extremely noisy due to antibodies' fast-evolving nature.

    “Because of antibodies' fast-evolving nature, MSAs on CDRs, especially in the CDR3 loops on the heavy chain of the antibodies which-despite being highly enriched in amino acid composition-are extremely noisy.”

  • The paper relies on this without support.

    Orphan proteins lack sequence and structure homology information.

    “Unlike antibodies, orphan proteins, by definition, lack sequence and structure homology information, and thus are also difficult to predict by MSA-based methods (Fig. 2B, Materials and Methods).”

  • The paper relies on this without support.

    Long-range protein contacts can be more accurately predicted by using the attention score from deeper layers compared to shallow layers.

    “More analysis of the ESM-1b model indicates that long-range protein contacts could be more accurately predicted by using the attention score from deeper layers compared to shallow layers.”

  • The paper relies on this without support.

    It is common that the length of proteins to be predicted is out of the range of the lengths of training samples.

    “However, it is common that the length of proteins to be predicted is out of the range of the lengths of training samples.”

  • The paper relies on this without support.

    AlphaFold2's 3D structure prediction is driven by triangle attention on edge representations and Invariant Point Attention in the Structure Module.

    “The 3D structure prediction of AlphaFold2 is driven by two important modules: triangle attention on edge representations and the Invariant Point Attention (IPA) in Structure Module.”

  • The paper relies on this without support.

    Triangle attention relies on the triangle multiplicative update on edge representations to enforce the edge representation to satisfy the triangle inequality of amino-acid distances, which significantly improves the feasibility of predicted structures based on MSA data.

    “The triangle attention relies on the”

  • The paper relies on this without support.

    Geometric consistency of AlphaFold2 is maintained on high-dimensional edge representations which later are integrated into node representations to generate the coordinate of each atom.

    “The geometric consistency of AlphaFold2 is maintained on the high-dimensional edge representations which later are integrated into node representations to generate the coordinate of each atom.”

  • The paper relies on this without support.

    Adjacent residues in a protein sequence are usually close in the 3D structure.

    “Adjacent residues in a protein sequence are usually closed in the 3D structure.”

  • The paper relies on this without support.

    Some protein chains need to interact with other biological entities such as RNAs and proteins to form a stable 3D structure.

    “One of the reasons is that a protein chain might need to interact with other biological entities such as RNAs and proteins to form a stable 3D structure.”

  • The paper relies on this without support.

    Some proteins have inherently hard-to-predict 3D structures because few homologous templates exist in the training data or the co-evolution information extracted from data is sparse.

    “There is another set of proteins for which 3D structures are inherently hard to predict as there are few homologous templates existing in the training data or the co-evolution information extracted from data is sparse.”

  • The paper relies on this without support.

    Discarding the experimentally resolved loss does not influence the folding performance.

    “We discard experimentally resolved loss which does not influence the folding performance.”

  • The paper relies on this without support.

    The protein language model includes more unlabeled training data than the 3D structure prediction model.

    “The protein language model and 3D structure prediction model are trained separately as the protein language model includes more unlabeled training data.”
13 of 13

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The unprecedented accuracy of AlphaFold2, RosettaFold, and others offers the opportunity to virtually screen any protein of interest with structure-based approach, irrespective of a priori ligandability.

    “The unprecedented accuracy of AlphaFold2,127 RosettaFold,128 and others129,130 offers, at least in principle, the opportunity to virtually screen any protein of interest with the structure-based approach, irrespective of its a priori ligandability (after AlphaFold2, the structural coverage of the human proteome increased from 48% to 76% of the total residues).131”

    From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [130]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • OmegaFold is described as both jointly and separately trained

    Major

    The main text says the entire model was jointly trained, but the training recipe says the protein language model and structure-prediction model were trained separately, with the language model frozen during structure-model training.

  • Ablation studies are attributed to the training-protocol table

    Minor

    The main text points to Table S3 for ablation studies, but Table S3 is labeled as the training protocol; the ablation results are labeled Table S4.

  • OmegaPLM both matches and doubles ESM-1b's layer count

    Minor

    The text says OmegaPLM uses the same number of layers as ESM-1b, then says its 66 layers double ESM-1b's layer count.

  • Last Geoformer layers are assigned conflicting primary effects

    Minor

    The main text says the remaining layers focus on reducing geometric inconsistency after contact accuracy plateaus. The figure caption instead says the last 30 layers mainly refine contact predictions, reporting a TopL accuracy increase from 0.578 to 0.704.

  • Attention activation is specified as both relu2 and softmax

    Minor

    The Gated Attention Unit description says the model replaces softmax with relu2, whereas the subsequent explanation says the authors chose softmax instead. Algorithm 1 also specifies softmax for the attention weights.

5 of 5

The citation lists haven’t been fetched yet

References 79 · Cited by 392

Research works printed but not linked: 16

  1. A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , Ł. U. Kaiser , I. Polosukhin , in Advances in Neural Information Processing Systems, I. Guyon , U. V. Luxburg , S. Bengio , H. Wallach , R. Fergus , S. Vishwanathan , R. Garnett , Eds. (Curran Associates, Inc., 2017; https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf), vol. 30.
  2. J. Devlin , M.-W. Chang , K. Lee , K. Toutanova , in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Association for Computational Linguistics, Minneapolis, Minnesota, 2019), pp. 4171–4186.
  3. T. Bepler , B. Berger , in International Conference on Learning Representations (2019; https://openreview.net/forum?id=SygLehCqtm).
  4. Crystallography: Protein data bank, 1971, Nat. New Biol, 233, 223
  5. 2022, Proceedings of Machine Learning Research, 162 of, 9099
  6. 2020, in Advances in Neural Information Processing Systems, 33, 1877
  7. 2019, Unified rational protein engineering with sequence-based deep representation learning, Nature Methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
  8. 2021, Learning the protein language: Evolution, structure, and function, Cell Systems, 12, 654, 10.1016/j.cels.2021.05.017 · DOI · Paper page
  9. 10.1126/science.abd7331 · DOI · Paper page
  10. 10.1038/S41586-021-03819-2 · DOI · Paper page
  11. A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , L. u. Kaiser , and I. Polosukhin , “Attention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon , U. V. Luxburg , S. Bengio , H. Wallach , R. Fergus , S. Vishwanathan , and R. Garnett , Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  12. P. J. Liu* , M. Saleh* , E. Pot , B. Goodrich , R. Sepassi , L. Kaiser , and N. Shazeer , “Generating wikipedia by summarizing long sequences,” in International Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=Hyg0vbWC-
  13. the UniProt Consortium, 2014, UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches, Bioinformatics, 31, 926
  14. 2013, Proceedings of the 30th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, 28, 1310
  15. 10.1038/nbt.3988 · DOI · Paper page
  16. A. Paszke , S. Gross , F. Massa , A. Lerer , J. Bradbury , G. Chanan , T. Killeen , Z. Lin , N. Gimelshein , L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.

Authors
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., Ma, J., & Peng, J.
Affiliations
  • Westlake University (CN)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
Corresponding author
  • Berger, Bonnie
  • Ma, Jianzhu
  • Peng, Jian
ORCID
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2022-07-22
DOI
10.1101/2022.07.21.500999
OpenAlex
W4286669150
Cited by
392
References
79
Citations by year
2026: 65 · 2025: 103 · 2024: 111 · 2023: 93 · 2022: 19 · 2013: 1
Type
posted content
Language
en
Open access
Licence
cc-by-nc-nd
Locations
bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function
Keywords
protein structure prediction, de novo structure prediction, single-sequence prediction, protein language models, protein folding, orphan proteins, antibody structure prediction, high-resolution protein structures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review