CoChat
Sign Up
PreprintCited by 5,666Protein Structure and DynamicsMachine Learning in BioinformaticsEnzyme Structure and Function

Evolutionary-scale prediction of atomic level protein structure with a language model

Open access
Lin, Zeming, Akin, Halil, Rao, Roshan, Hie, Brian, Zhu, Zhongkai, Lu, Wenting, Smetanin, Nikita, Verkuil, Robert +7 more·2022·Science
DOI 10.1101/2022.07.20.500902
Open this paper in CoChat — free account

Abstract

Artificial intelligence has the potential to open insight into the structure of proteins at the scale of evolution. It has only recently been possible to extend protein structure prediction to two hundred million cataloged proteins. Characterizing the structures of the exponentially growing billions of protein sequences revealed by large scale gene sequencing experiments would necessitate a break-through in the speed of folding. Here we show that direct inference of structure from primary sequence using a large language model enables an order of magnitude speed-up in high resolution structure prediction. Leveraging the insight that language models learn evolutionary patterns across millions of sequences, we train models up to 15B parameters, the largest language model of proteins to date. As the language models are scaled they learn information that enables prediction of the three-dimensional structure of a protein at the resolution of individual atoms. This results in prediction that is up to 60x faster than state-of-the-art while maintaining resolution and accuracy. Building on this, we present the ESM Metage-nomic Atlas. This is the first large-scale structural characterization of metagenomic proteins, with more than 617 million structures. The atlas reveals more than 225 million high confidence predictions, including millions whose structures are novel in comparison with experimentally determined structures, giving an unprecedented view into the vast breadth and diversity of the structures of some of the least understood proteins on earth.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces ESM-2 protein language models up to 15B parameters and ESMFold, an end-to-end single-sequence structure predictor that reaches atomic-resolution accuracy while being up to 60x faster than state-of-the-art pipelines. It uses ESMFold to build the ESM Metagenomic Atlas, folding over 617 million MGnify90 sequences and identifying more than 225 million high-confidence structures, many novel relative to known sequences and structures.

Key findings

  • Scaling ESM-2 from 8M to 15B parameters improves language modeling and structural information: validation perplexity falls from 10.45 to 6.37, unsupervised long-range contact precision (LR P@L) rises from 0.16 to 0.54, and structure-module-only LDDT rises from 0.48 to 0.72. The 15B model reaches TM-score 0.71 on CAMEO and 0.54 on CASP14, 0.064 higher than the 150M ESM-2 model on both.
  • ESMFold achieves average TM-score 0.83 on CAMEO and 0.68 on CASP14, compared with AlphaFold2 at 0.88 and 0.85 and RosettaFold at 0.82 on CAMEO. On a single NVIDIA V100 GPU, ESMFold predicts a 384-residue protein in 14.2 seconds, 6x faster than a single AlphaFold2 model, with up to ~60x speedup on shorter sequences; MSA search alone can take over 10 minutes.
  • The ESM Metagenomic Atlas folds over 617 million MGnify90 sequences in 2 weeks on ~2,000 GPUs, yielding ~365 million good-confidence predictions (mean pLDDT > 0.5 and pTM > 0.5) and ~225 million high-confidence predictions (mean pLDDT > 0.7 and pTM > 0.7). In a 1-million high-confidence sample, 76.8% have <90% sequence identity to UniRef90, 12.6% have no PDB structure with TM-score > 0.5, and 10.4% have both low structural similarity (TM-score ≤ 0.5) and no close sequence homolog (<30% identity).
  • Confidence is well calibrated: for high-confidence predictions (pLDDT > 0.7), CAMEO accuracy is comparable to AlphaFold2 (ESMFold LDDT = 0.83, AlphaFold2 LDDT = 0.85); for very high confidence (pLDDT > 0.9), median all-atom RMSD95 is 1.42 Å and backbone RMSD95 is 0.94 Å.

Methods

  • ESM-2 is a BERT-style transformer masked language model trained on UniRef50/UniRef90 sequences, scaled from 8M to 15B parameters, with models trained for up to 500K updates (15B for 270K updates) and evaluated on held-out UniRef50 clusters.
  • ESMFold adds a 48-block folding head to a frozen 3B-parameter ESM-2, trained on ~25K PDB clusters covering ~325K experimentally determined chains plus ~12M AlphaFold2-predicted structures; evaluation uses temporally held-out CAMEO (194 proteins) and CASP14 (51 targets) test sets.
  • Metagenomic analysis folded MGnify90 sequences of length 20–1024, covering 99% of the database, and used Foldseek, MMseqs2, blastp, and UMAP for structural and sequence similarity analyses.

Limitations

  • The paper states that ESMFold was not trained on protein complexes, so complex prediction is out of distribution; it also notes that ablating MSAs for AlphaFold and RosettaFold is an artificial single-sequence setting because those models were not explicitly trained for single sequences.
  • The paper states that calibration becomes critical as structure prediction scales, and that current models are far from the limit of scale in parameters, sequence data, and compute.
  • Evident from the text: this is a bioRxiv preprint not certified by peer review; ESMFold is less accurate than AlphaFold2 overall on CAMEO (0.83 vs 0.88 TM-score) and CASP14 (0.68 vs 0.85), with a larger gap on hard CAMEO targets (0.45 vs 0.62).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2022.07.20.500902

Cites

86 of 86 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 16 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 2

    76.8% of high confidence predictions are separate from UniRef90 by at least 90% sequence identity.

    “the vast majority (76.8%) of high confidence predictions being separate from UniRef90 (32) by at least 90% sequence identity”
    Quote in the paperSubstantial claim1 of 1 sources checked
  • Interpretationcitation
    § 6. Conclusions · p. 9

    Very high confidence predictions in the metagenomic atlas are expected to often be reliable at a resolution sufficient for insight similar to experimentally determined structures, such as into the biochemistry of active sites.

    “Very high confidence predictions in the metagenomic atlas are expected to often be reliable at a resolution sufficient for insight similar to experimentally determined structures, such as into the biochemistry of active sites (61);”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [61] source

      The High-resolution models section says predicted models can meet demanding structural requirements and sometimes support ligand-binding studies; the Medium-resolution models section says predictions can help locate active sites. The paper does not assess predictions in the metagenomic atlas or establish that its very high-confidence predictions are often reliable enough to provide experimentally comparable insight into active-site biochemistry. Page numbers are unavailable in our copy. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Findingcitation
    § 4. Evolutionary-scale structural characterization of metagenomics · p. 7

    The paper folds over 617 million sequences from the MGnify90 database.

    “We fold over 617 million sequences from the MGnify90 database (25).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] source

      The 'UPDATES TO THE MGnify PROTEIN DATABASE' section describes MGnify protein sequences clustered at 90% identity, supporting the identification of the database. It reports over 1.1 billion sequences in the full database, but does not give a count of 617 million for the 90%-clustered set. The citing authors' folding of those sequences is their own work, not a finding attributed to this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 7

    In a comparison with AlphaFold-Multimer on 2,978 recent multimeric complexes, ESMFold achieves the same qualitative DockQ categorization for 53.2% of chain pairs.

    “In a comparison with AlphaFold-Multimer (45) on a dataset of 2,978 recent multimeric complexes deposited in the PDB, ESMFold achieves the same qualitative DockQ (39) categorization for 53.2% of chain pairs, despite not being trained on protein complexes (Fig. S6).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [45] Richard Evans , Michael O’Neill , Alexander Pritzel , Natasha Antropova , Andrew Senior , Tim Green , Augustin Žídek , Russ Bates , Sam Blackwell , Jason Yim , Olaf Ronneberger , Sebastian Bodenstein , Michal Zielinski , Alex Bridgland , Anna Potapenko , Andrew Cowie , Kathryn Tunyasuvunakool , Rishub Jain , Ellen Clancy , Pushmeet Kohli , John Jumper , and Demis Hassabis . Protein complex prediction with AlphaFold-Multimer, March 2022. URL https://www.biorxiv.org/content/10.1101/2021.10.04.463034v2. Pages: 2021.10.04.463034 Section: New Results. source
    • cites [39] source
  • Findingreasoning
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 4

    Training ESM-2 results in the emergence of structure in the models.

    “This training also results in the emergence of structure in the models.”
    Quote in the paperSubstantial claim
  • May combine two claims. 32
  • Classified as probably not a substantial claim. 27
  • The quote only partly states this claim. 1
  • Borderline: one of the tests for a substantial claim is unclear. 1

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    The ESM Metagenomic Atlas is the first large-scale structural characterization of metagenomic proteins, with more than 617 million structures, folding practically all sequences in MGnify90.

    “This is the first large-scale structural characterization of metagenomic proteins, with more than 617 million structures.”
    • Also stated in: § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 2 · plain
      “We present the first evolutionary scale structural characterization of a metagenomic resource, folding practically all sequences in MGnify90 (25), over 617M proteins.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] source

      The ‘UPDATES TO THE MGnify PROTEIN DATABASE’ section describes more than 1.1 billion protein sequences and their clustering, not predicted structures. Searches for the ESM Metagenomic Atlas, 617 million structures, and protein folding found no passage addressing the claimed structural characterization. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § A.1.3. STRUCTURE VALIDATION AND TEST SETS · p. 17

    The second test set consists of 51 targets from the CASP14 competition.

    “Our second test set consists of 51 targets from the CASP14 competition (37).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [37] source

      The Introduction describes CASP14 modeling targets, supporting their use as a source for a test set. It reports 68 tertiary-structure modeling targets, but does not identify the citing authors’ 51-target selection or verify its size. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 4. Evolutionary-scale structural characterization of metagenomics · p. 7

    MGnify sequence MGYP000936678158 has no significant sequence matches to any entry in UniRef90, nor any significant matches via a jackhmmer reference proteome search, but has a predicted structure conserved across many nucleases (PDB 5YET B, TM-score 0.68; PDB 3HR4 A, TM-score 0.67).

    “MGnify sequence MGYP000936678158 has no significant sequence matches to any entry in UniRef90, nor any significant matches via a jackhmmer (47) reference proteome search, but has a predicted structure conserved across many nucleases (PDB 5YET B, TM-score 0.68; PDB 3HR4 A, TM-score 0.67)”
    Quote in the paperSubstantial claim1 of 1 sources checked

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § A.1.3. STRUCTURE VALIDATION AND TEST SETS · p. 17

    During method development, a temporally held out validation set from CAMEO (August 2021 to January 2022) was used.

    “During method development (e.g. hyperparameter selection), we used a temporally held out validation set obtained from the Continuous Automated Model EvaluatiOn (CAMEO) server (36) by filtering from August 2021 to January 2022.”
    Quote in the paperNot a substantial claim1 of 1 sources checked

  • May combine two claims.

    Findingown data
    § 2. Atomic resolution structure emerges in language models trained on protein sequences · p. 2

    The ESM-2 model family significantly outperforms ESM-1b at comparable parameters and outperforms other recent protein language models on structure prediction benchmarks.

    “The resulting ESM-2 model family significantly outperforms previously state-of-the-art ESM-1b (a ∼ 650 million parameter model) at a comparable number of parameters, and on structure prediction benchmarks it also outperforms other recent protein language models (Table S3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 4

    At 270K training steps, the 8M parameter model has a perplexity of 10.45, and the 15B model reaches a perplexity of 6.37.

    “At 270K training steps the 8M parameter model has a perplexity of 10.45, and the 15B model reaches a perplexity of 6.37”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § Abstract · p. 1

    ESM-2 language model prediction is up to 60x faster than state-of-the-art while maintaining resolution and accuracy.

    “This results in prediction that is up to 60x faster than state-ofthe-art while maintaining resolution and accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 2

    The new approach eliminates the need for a multiple sequence alignment and greatly simplifies the neural architecture used for inference.

    “This removes costly aspects of current state-of-the-art structure prediction pipelines, eliminating the need for a multiple sequence alignment, while at the same time greatly simplifying the neural architecture used for inference.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 4

    Attention patterns develop in ESM-2 that correspond to tertiary structure, and scaling leads to large improvements in the understanding of structure.

    “Attention patterns develop in ESM-2 that correspond to tertiary structure (Fig. 1A), and scaling leads to large improvements in the understanding of structure (Fig. 1B).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4. Evolutionary-scale structural characterization of metagenomics · p. 7

    The characterization produces ∼225 million predictions with high confidence (mean pLDDT > 0.7 and pTM > 0.7), corresponding to ∼36% of total structures folded.

    “and ∼ 225 million predictions with high confidence (mean pLDDT &gt; 0 . 7 and pTM &gt; 0 . 7 ) corresponding to ∼ 36% of total structures folded (Fig. 3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.1.3. STRUCTURE VALIDATION AND TEST SETS · p. 17

    The first test set consists of all 194 CAMEO test proteins from April 01, 2022 through June 25, 2022.

    “The first test is from CAMEO, consisting of all 194 test proteins from April 01, 2022 through June 25, 2022.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 6

    This results in a speed improvement for structure prediction of more than an order of magnitude while maintaining high resolution accuracy.

    “This results in a speed improvement for structure prediction of more than an order of magnitude while maintaining high resolution accuracy.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data
    § 4. Evolutionary-scale structural characterization of metagenomics · p. 7

    Predictions were completed in 2 weeks on a cluster of approximately 2,000 GPUs.

    “We were able to complete the predictions in 2 weeks on a cluster of approximately 2,000 GPUs (Appendix A.4.1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 6

    On CAMEO, language model perplexity has a Pearson correlation of -0.55 with TM-score; on CASP14, the correlation is -0.67.

    “On the CAMEO test set, language model perplexity has a Pearson correlation of -0.55 with the TM-score between the predicted and experimental structures; on CASP14, the correlation is -0.67 (Fig. 2B).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § A.1.2. STRUCTURE TRAINING SETS FOR ESMFOLD · p. 17

    The ESMFold training set consists of 25,450 clusters covering a total of 325,498 chains.

    “This results in 25,450 clusters covering a total of 325,498 chains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 6

    ESMFold achieves competitive accuracy with RosettaFold on CAMEO, which averages a TM-score of 0.82.

    “ESMFold achieves competitive accuracy with RosettaFold on CAMEO, which averages a TM-score of 0.82.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § A.2.1. COMPUTING UNSUPERVISED CONTACT PREDICTION FROM LANGUAGE MODELS · p. 18

    The average long range P@L across 20 bootstrapped samples of 20 training proteins was 0.4287 with a standard deviation of 0.0028.

    “The average long range P@L was 0.4287 with a standard deviation of 0.0028.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4. Evolutionary-scale structural characterization of metagenomics · p. 7

    MGnify sequence MGYP004000959047 has no significant UniRef90 or jackhmmer reference proteome matches but its predicted structure has high similarity to experimental structures of lipid binding domains (PDB 6BYM A, TM-score 0.80; PDB 5YQP B, TM-score 0.78).

    “MGnify sequence MGYP004000959047 has no significant UniRef90 or jackhmmer reference proteome matches but its predicted structure has high similarity to experimental structures of lipid binding domains (PDB 6BYM A, TMscore 0.80; PDB 5YQP B, TM-score 0.78)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 6

    AlphaFold2 achieves 0.88 and 0.85 on CAMEO and CASP14 respectively.

    “Using the search protocols released with AlphaFold2, including MSAs and templates, AlphaFold2 achieves 0.88 and 0.85 on CAMEO and CASP14 respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.1.2. STRUCTURE TRAINING SETS FOR ESMFOLD · p. 17

    The authors generated a set of 13,477,259 structure predictions with AlphaFold2.

    “For Hsu et al. (62) we generated a set of 13,477,259 structure predictions with AlphaFold2 using MSAs generated via the process in Rao et al. (40).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 4

    The 15 billion parameter model reaches a TM-score of 0.71 on the CAMEO test set and 0.54 on the CASP14 test set, 0.064 points higher than the 150 million parameter ESM-2 model on both.

    “The 15 billion parameter model reaches a TM-score of 0.71 on the CAMEO test set and 0.54 on the CASP14 test set, 0.064 points higher than the 150 million parameter ESM-2 model on both (Fig. 1E).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 6. Conclusions · p. 9

    The structural characterization reveals the structures of hundreds of millions of proteins that were previously unknown, millions of which are novel compared to experimentally determined structures.

    “This characterization reveals the structures of hundreds of millions proteins that have been previously unknown, millions of which are novel in comparison to experimentally determined structures.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 6

    ESMFold achieves an average TM-score of 0.83 on CAMEO and 0.68 on CASP14.

    “ESMFold achieves an average TM-score of 0.83 on CAMEO and 0.68 on CASP14.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § A.1.3. STRUCTURE VALIDATION AND TEST SETS · p. 17

    ESMFold is able to make predictions on all sequences, including the length-2166 target T1044.

    “No filtering is performed on these test sets, as ESMFold is able to make predictions on all sequences, including the length-2166 target T1044.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § A.1.2. STRUCTURE TRAINING SETS FOR ESMFOLD · p. 17

    Only 1.5% of the AlphaFold2-generated structure prediction dataset is removed by the mean pLDDT > 70 filter.

    “Because of the way the dataset is constructed, only 1.5% of the dataset is removed with this filter.”
    Quote in the paperNot a substantial claim

  • Borderline on S4 (based): unclear from the text.

    Findingnone given
    § 3. Accelerating accurate atomic resolution structure prediction with a language model · p. 4

    Language models greatly accelerate state-of-the-art high resolution structure prediction.

    “Language models greatly accelerate state-of-the-art high resolution structure prediction.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 6

    On a single NVIDIA V100 GPU, ESMFold makes a prediction on a protein with 384 residues in 14.2 seconds, 6x faster than a single AlphaFold2 model.

    “On a single NVIDIA V100 GPU, ESMFold makes a prediction on a protein with 384 residues in 14.2 seconds, 6x faster than a single AlphaFold2 model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 4

    Across all models, there is a correlation of -0.99 between validation perplexity and CASP14 TM-score, and -1.00 between validation perplexity and CAMEO TM-score.

    “Across all models (Table S3) there is a correlation of -0.99 between validation perplexity and CASP14 TM-score, and -1.00 between validation perplexity and CAMEO TM-score”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 7

    On the CAMEO test set, ESMFold predictions have a median all-atom RMSD 95 of 1.91 Å and backbone RMSD 95 of 1.33 Å.

    “On the CAMEO test set, ESMFold predictions have a median all-atom RMSD 95 (root-mean-squared deviation at 95% residue coverage) of 1.91 ˚ A and backbone RMSD 95 of 1.33 ˚ A.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 24

    For RoseTTAfold, the speed of the SE(3) Transformer dominates, especially at low sequence lengths.

    “For RoseTTAfold, the speed of the SE(3) Transformer dominates, especially at low sequence lengths.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § A.1.5. RECENT-PDB-MULTIMERS · p. 18

    The filtering steps for the Recent-PDB-Multimers dataset result in a total of 2978 clusters.

    “These steps result in a total of 2978 clusters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 24

    At low sequence lengths, ESMFold is dominated by language model performance, while the O(N^3) computation of pairwise representations takes over at high sequence lengths.

    “At low sequence lengths, ESMFold is dominated by language model performance, while the O ( N 3 ) computation of pairwise representations takes over at high sequence lengths.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 28

    Differentiation is greater in the 'medium' and 'hard' classes, which have lower MSA depth and are better predicted by AlphaFold2.

    “Differentiation is greater in the ªmediumº and ªhardº classes, which have lower MSA depth and are better predicted by AlphaFold2.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 24

    ESMFold has an over 60x speed advantage for shorter protein sequences, and a reasonable speed advantage for longer protein sequences.

    “We see an over 60x speed advantage for shorter protein sequences, and a reasonable speed advantage for longer protein sequences.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.4.1. FOLDING 620 MILLION SEQUENCES FROM MGNIFY · p. 21

    The entire 620 million sequences were folded in 2 weeks time.

    “which we were able to do in 2 weeks time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A.2.5. ESM-2 ABLATION EXPERIMENTS · p. 19

    Both RoPE and newer data significantly improve unsupervised contact prediction results.

    “Unsupervised contact prediction results show that both RoPE and newer data significantly improve the results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § A.4.1. FOLDING 620 MILLION SEQUENCES FROM MGNIFY · p. 21

    On a homogeneous network GPU cluster of V100s, folding the entire 620 million sequences would take approximately 28,000 GPU days.

    “We estimate that on a homogeneous network GPU cluster of V100s, the entire 620 million sequences would take approximately 28,000 GPU days to fold”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 24

    Language model perplexity and ESMFold pLDDT are both well correlated with actual structure prediction accuracy on CASP14 and CAMEO.

    “Language model perplexity and ESMFold pLDDT are both well correlated with actual structure prediction accuracy on CASP14 and CAMEO.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 23

    After 270K updates the 8M parameter model has a perplexity of 10.45, and the 15B model reaches a perplexity of 6.37.

    “After 270K updates the 8M parameter model has a perplexity of 10.45, and the 15B model reaches a perplexity of 6.37.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § A.5. Orphan Proteins · p. 22

    After filtering, there are 104 sequences with MSA depth ≤100, 70 sequences with MSA depth ≤10, and 22 sequences with MSA depth =1.

    “After filtering, there are 104 sequences with MSA depth ≤ 100 , 70 sequences with MSA depth ≤ 10 , and 22 sequences with MSA depth = 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.4.2. ANALYSIS OF FOLDED METAGENOMICS STRUCTURES · p. 21

    The high-confidence subset with mean pLDDT greater than 0.9 corresponds to approximately 59K structures.

    “analyzed the high-confidence subset with mean pLDDT greater than 0.9, corresponding to ∼ 59K structures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § A.1.5. RECENT-PDB-MULTIMERS · p. 18

    The evaluation on the Recent-PDB-Multimers dataset results in a total of 3505 unique chain pairs.

    “This results in a total of 3505 unique chain pairs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § A.2.5. ESM-2 ABLATION EXPERIMENTS · p. 19

    Sampling from UniRef90 clusters causes a slight regression in unsupervised contact prediction results.

    “We do observe a slight regression when sampling from UniRef90 clusters”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1. Introduction · p. 1

    Language model capabilities develop as a function of scale, with greater capabilities emerging as computation, data, and number of parameters increase.

    “These capabilities develop as a function of scale, with greater capabilities emerging as computation, data, and number of parameters increase.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites 19±22

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § 1. Introduction · p. 1

    Within a protein family, structure and function can be inferred from the patterns in sequences.

    “Within a protein family, structure and function can be inferred from the patterns in sequences (4, 5).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [4] source
    • cites [5] 2008 source

  • May combine two claims.

    Interpretationreasoning
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 4

    Improvement on sequences with high evolutionary depth saturates at lower model scales, and improvement on sequences with low evolutionary depth continues as models increase in size.

    “This means that improvement on sequences with high evolutionary depth saturates at lower model scales, and improvement on sequences with low evolutionary depth continues as models increase in size.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1. Introduction · p. 1

    Language models of text develop emergent capabilities connected to the underlying meaning of the text.

    “language models of text, despite the simplicity of their training objectives, such as filling in missing words or predicting the next word, are shown to develop emergent capabilities that are connected to the underlying meaning of the text.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites 19±22

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data
    § Abstract · p. 1

    As protein language models are scaled, they learn information that enables prediction of the three-dimensional structure of a protein at the resolution of individual atoms, and atomic resolution structure prediction emerges and continues to improve over four orders of magnitude in parameter scale.

    “As the language models are scaled they learn information that enables prediction of the three-dimensional structure of a protein at the resolution of individual atoms.”
    • Also stated in: § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 2 · plain
      “We discover that atomic resolution structure prediction emerges and continues to improve in language models over the four orders of magnitude in parameter scale.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § A.1.3. STRUCTURE VALIDATION AND TEST SETS · p. 17

    No filtering is performed on the test sets.

    “No filtering is performed on these test sets, as ESMFold is able to make predictions on all sequences, including the length-2166 target T1044.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § A.1.1. SEQUENCE DATASET USED TO TRAIN ESM-2 · p. 17

    The training set had sequences removed that matched a validation sequence with 50% sequence identity.

    “All train sequences which match a validation sequence with 50% sequence identity under this search are removed from the train set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § A.1.3. STRUCTURE VALIDATION AND TEST SETS · p. 17

    The two test sets were both chosen to be temporally held out from the supervised training set.

    “We report results by testing 3D structure prediction models on two test sets, both chosen to be temporally held out from our supervised training set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 25

    All compared sequences in the orphan protein analysis are temporally held out from the training set.

    “All compared sequences are temporally held out from the training set.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § A.2.5. ESM-2 ABLATION EXPERIMENTS · p. 19

    The difference from UniRef90 cluster sampling is small and likely to help for larger models.

    “however we believe this difference is small and the UniRef90 cluster sampling is likely to help for the larger models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 24

    The number of SE(3) max-iterations are artificially limited to 20 (default 200) and no MSAs are used as input for these measurements.

    “The number of SE(3) max-iterations are artificially limited to 20 (default 200) and no MSAs are used as input for these measurements.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 24

    This only measures the network forward time, and does not include the time taken to compute sidechains with PyRosetta or search for MSAs.

    “Additionally, this only measures the network forward time, and does not include the time taken to compute sidechains with PyRosetta or search for MSAs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A.2.4. TRAINING ESM-2 · p. 19

    BERT style models are amenable to very large batch sizes.

    “Since BERT style models have been shown to be amenable to very large batch sizes (67), we increased our effective batch size to 2M tokens.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [67] Yang You , Jing Li , Sashank Reddi , Jonathan Hseu , Sanjiv Kumar , Srinadh Bhojanapalli , Xiaodan Song , James Demmel , Kurt Keutzer , and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes. page 38, 2020.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § Evolutionary-scale prediction of atomic level protein structure with a language model · p. 25

    ESMFold is not trained on protein complexes.

    “even though ESMFold is not trained on protein complexes.”
    Quote in the paperSubstantial claim
57 of 57

  • The paper relies on this without support.

    AlphaFold2 predictions provide a valid reference for assessing the accuracy of ESMFold predictions on metagenomic proteins.

    “We also assess calibration against AlphaFold predictions on metagenomic proteins.”
    “On a random subset of ∼ 4K metagenomic sequences, there is a high correlation (Pearson r = 0.79) between ESMFold pLDDT and the LDDT to AlphaFold2 predictions (Fig. 3A).”

  • The paper relies on this without support.

    The MGnify90 database is representative of metagenomic protein diversity.

    “We fold over 617 million sequences from the MGnify90 database (25).”

  • The paper relies on this without support.

    The PDB database provides a comprehensive reference for experimentally determined protein structures.

    “no structure is found with TM-score over 0.5 in the PDB database”
    “no experimentally determined structure with a similar fold could be identified”

  • The paper relies on this without support.

    Absolute positional encoding methods don't extrapolate well beyond the context window they are trained on.

    “However, absolute positional encoding methods don't extrapolate well beyond the context window they are trained on.”

  • The paper relies on this without support.

    Orphan proteins are very challenging for current structure prediction models due to a lack of evolutionary information.

    “Due to a lack of evolutionary information, these sequences can be very challenging for current structure prediction models.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Evo reaches performance competitive with leading protein-specific language models on bacterial proteins.

    “Evo also reaches competitive performance with leading protein-specific language models (41, 46–48) (Fig. 2B).”

    From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [47]

  • AI modeling paradigms have been used successfully in the biomolecular realm, for example, to predict 3D molecular structures from sequences and interactions between different molecular components.

    “Such modeling paradigms have been used successfully in the biomolecular realm, for example, to predict three-dimensional (3D) molecular structures from sequences32–34 and interactions between different molecular components.35–38”

    From How to build the virtual cell with artificial intelligence: Priorities and opportunities (2024) · cites this paper as [34]

  • Machine-learning tools such as AF2, RoseTTAFold, and ESMFold are currently the most accurate answer to the structure prediction problem.

    “are the best and most accurate answer to the structure prediction problem to date”

    From The power and pitfalls of AlphaFold2 for structure prediction beyond rigid globular proteins (2024) · cites this paper as [19]

  • The unprecedented accuracy of AlphaFold2, RosettaFold, and others offers the opportunity to virtually screen any protein of interest with structure-based approach, irrespective of a priori ligandability.

    “The unprecedented accuracy of AlphaFold2,127 RosettaFold,128 and others129,130 offers, at least in principle, the opportunity to virtually screen any protein of interest with the structure-based approach, irrespective of its a priori ligandability (after AlphaFold2, the structural coverage of the human proteome increased from 48% to 76% of the total residues).131”

    From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [129]

  • The repository of publicly available structures, consisting of the combined databases of AFDB and the ESMatlas, is approaching a billion entries.

    “This difficulty is amplified by the fact that the repository of publicly available structures, consisting of the combined databases of AFDB and the ESMatlas38, is approaching a billion entries.”

    From Clustering predicted structures at the scale of the known protein universe (2023) · cites this paper as [38]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting PDB identifier for the nuclease example

    Minor

    The main text identifies the comparison structure as PDB 3HR4 A, but the Figure 4 caption identifies it as PDB 3H4R. These are different identifiers for the same described example.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , Łukasz Kaiser , and Illia Polosukhin . Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017. URL https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf.
  2. 2022, Chain of Thought Prompting Elicits Reasoning in Large Language Models
  3. Thanumalayan Sankaranarayana Pillai, 2022, PaLM: Scaling Language Modeling with Pathways
  4. 2021, RoFormer: Enhanced Transformer with Rotary Position Embedding
  5. 2020, UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

Cited by

Sorted by citations

Related

Authors
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., & Rives, A.
Affiliations
  • New York University (US)
  • Stanford University (US)
Corresponding author
Rives, Alexander
ORCID
Venue
Science
Publisher
openRxiv
Published
2022-07-21
Volume
379
Issue
6637
Pages
1123-1130
ISSN
0036-8075 (ISSN-L)
DOI
10.1101/2022.07.20.500902
OpenAlex
W4327550249
Cited by
5,523
References
71
Citations by year
2026: 1885 · 2025: 1849 · 2024: 1221 · 2023: 530 · 2022: 30 · 2020: 1 · 2013: 1 · 2012: 3
FWCI
413.97
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Science (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function
Keywords
protein structure prediction, atomic resolution structure, large language models, Earth system models, protein sequence representation, evolutionary information, protein diversity
MeSH terms
Machine Learning, Amino Acid Sequence, Protein Conformation, Proteins, Evolution, Molecular, Sequence Analysis, Protein

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review