The DOI matches its Crossref record; 130 of the 133 references it cites were found in a registry or an index; 6 of the 53 sources its claims cite were checked for support.
Learning the protein language: Evolution, structure, and function
Open accessAbstract
Language models have recently emerged as a powerful machine-learning approach for distilling information from massive protein sequence databases. From readily available sequence data alone, these models discover evolutionary, structural, and functional organization across protein space. Using language models, we can encode amino-acid sequences into distributed vector representations that capture their structural and functional properties, as well as evaluate the evolutionary fitness of sequence variants. We discuss recent advances in protein language modeling and their applications to downstream protein property prediction problems. We then consider how these models can be enriched with prior biological knowledge and introduce an approach for encoding protein structural knowledge into the learned representations. The knowledge distilled by these models allows us to improve downstream function prediction through transfer learning. Deep protein language models are revolutionizing protein biology. They suggest new ways to approach protein and therapeutic design. However, further developments are needed to encode strong biological priors into protein language models and to increase their accessibility to the broader community.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Protein language models learn evolutionary, structural, and functional organization from large sequence databases, and adding structure supervision improves downstream prediction through transfer learning. The authors present a multi-task LSTM trained on 76 million sequences plus 28,000 structures and show gains in structural similarity, transmembrane prediction, and sequence-to-phenotype prediction.
Key findings
- A multi-task LSTM (MT-LSTM) trained with masked language modeling, residue-residue contact prediction, and SCOP structural similarity prediction learned embeddings that “dramatically outperform” sequence comparison methods and even outperform TMalign for structural similarity; structure supervision was critical for fine-grained organization, and MT-LSTM outperformed the prior SSA-LSTM.
- For transmembrane region prediction, BiLSTM+CRF models using MT-LSTM embeddings significantly outperformed DLM-LSTM embeddings (paired t-test, p=0.044) and also outperformed existing predictors and SSA-LSTM.
- For sequence-to-phenotype prediction on deep mutational scanning datasets, MT-LSTM embeddings with Gaussian process regression achieved an average Spearman correlation of 0.65, significantly outperforming ECNet’s 0.60 (paired t-test, p=0.006).
- The paper notes that language models predicted SARS-CoV-2 spike mutations with high antibody-escape potential, but as of publication viral escape had not yet been experimentally verified.
Methods
- The authors trained a three-layered bidirectional LSTM with skip connections and 1024 hidden units in each direction per layer. The MT-LSTM combined three losses: masked language modeling (10% of amino acids randomly masked), residue-residue contact prediction (contacts defined as residues within 8 Å), and SCOP structural similarity prediction via ordinal regression; MT-LSTM loss weights were λmasked=0.5, λcontact=0.9, λsimilarity=0.1, while DLM-LSTM used only λmasked=1.
- Training data were 76,215,872 UniRef90 sequences (July 2018) and 28,010 SCOPe ASTRAL 2.06 protein sequences with structures, split into 22,408 training and 5,602 testing sequences. Models were trained for 1,000,000 parameter updates with Adam at learning rate 0.0001 on a single NVIDIA V100 GPU; training took roughly 13 days for DLM-LSTM and 51 days for MT-LSTM.
- Evaluations included structural similarity on 100,000 randomly sampled SCOPe ASTRAL test pairs, transmembrane prediction on the TOPCONS2 dataset of 5,154 proteins using 10-fold cross validation, and sequence-to-phenotype prediction on deep mutational scanning datasets using 5-fold cross validation and Gaussian process regression with an RBF kernel.
Limitations
- The paper states that further developments are needed to encode strong biological priors into protein language models and to increase their accessibility to the broader community; it also states that current models make heavily simplified phylogenetic assumptions, treating sequences as independent draws from a star phylogeny.
- The paper states practical limitations including high compute costs, murky design guidelines, ad hoc model development, quadratic scaling of transformers with sequence length (requiring long sequences to be excluded or truncated), and poor learning of long-range dependencies by RNNs.
- Evident from the text: the new results come from a single MT-LSTM architecture and specific benchmarks, and the SARS-CoV-2 escape predictions had not been experimentally verified as of publication.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
21 Not checked yet · 11 Own result · 9 Own interpretation · 2 Supported so far · 2 Sources not found · 1 Partly supported · 1 Own method · 1 No basis given · accuracy not yet measured
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“New work has since shown that the representations learned by language models are also powerful features for learning of variant fitness as a subsequent supervised learning task 32,34, building on earlier observations that language models can improve protein property prediction through transfer learning 27.”
§ Learning the Protein Language: Evolution, Structure and Function > Protein language models distill information from massive protein sequence databases
What the sources say
- Supportedcites [32] Rives A Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv (2019). PMC8053943 33876751 source
The Prediction of Mutational Effects section (Fig. 7; page not provided) reports supervised fine-tuning of learned Transformer representations on mutational data and says they outperform LSTM representations on two datasets. The Discussion also says fine-tuning matches state-of-the-art results for variant activity prediction. (LLM, conf=0.94)
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When trained on tens of thousands of evolutionarily related proteins, the learned probability mass function describing the empirical distribution over naturally occurring sequences has shown promise for predicting the fitness of sequence variants 29-31.”
§ Learning the Protein Language: Evolution, Structure and Function > Protein language models distill information from massive protein sequence databases
What the sources say
- Not checked yetcites [29] Riesselman AJ, Ingraham JB & Marks DS Deep generative models of genetic variation capture the effects of mutations. Nat. Methods 15, 816– 822 (2018). 30250057 PMC6693876 source
- Not checked yetcites [30] Hie B, Zhong E, Bryson B & Berger B Learning mutational semantics. Adv. Neural Inf. Process. Syst 33, (2020).
- Partly supportedcites [31] Hie B, Zhong ED, Berger B & Bryson B Learning the language of viral evolution and escape. Science 371, 284– 288 (2021). 33446556 sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also find that increasing model size improves transfer learning performance.”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this organization is improved when we include structure supervision in the language model training (Figure 4b).”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, representations learned by our MT-LSTM model significantly outperform (paired t-test, p=0.044) the embeddings learned by our DLM-LSTM model on this application (Figure 5b).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Furthermore, the multi-task learning approach outperforms a two-step learning approach presented previously (SSA-LSTM)27.”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because we embed sequences into a semantic representation space, we can find structurally related proteins even though their sequences are not closely related (Figure 4c, Supplemental Table 1).”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that even though the DLM-LSTM model was trained using only sequence information, protein sequences still organize roughly by structure in embedding space.”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because TMscore is not symmetric, we calculate TMscore for both comparison directions and average them together for each protein pair. We found this outperformed other methods of combining the two scores.”
§ Learning the Protein Language: Evolution, Structure and Function > Methods > Protein structural similarity prediction evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The key insight was that the sequence representations (vector embeddings) learned by the language model were powerful features for solving other prediction problems.”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
- Probably not a substantial claim. 8
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“They also learn summary representations, powerful features that can be used to better capture sequence relationships and link sequence to function via transfer learning 27,28,32-35.”
§ Learning the Protein Language: Evolution, Structure and Function > Introduction
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
- Supportedcites [28] Alley EC, Khimulya G, Biswas S, AlQuraishi M & Church GM Unified rational protein engineering with sequence-based deep representation learning. Nat. Methods 16, 1315– 1322 (2019). 31636460 PMC7067682 source
The Results section reports that UniRep representations group proteins by structural, evolutionary and functional similarity. The Methods sections on computing representations and supplemental generalization describe sequence-summary vectors used for function prediction and evaluate transfer to proteins outside the training set. No page numbers are available in our copy. (LLM, conf=0.91)
- Supportedcites [32] Rives A Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv (2019). PMC8053943 33876751 source
In “Biological Variations Are Encoded in Representation Space” (page unavailable), the paper describes a summary vector for each protein and reports that related genes cluster in the learned representation space. The Discussion (page unavailable) says these learned features can be used for supervised prediction directly from sequence, including variant activity prediction. (LLM, conf=0.92)
- Supportedcites [33] Rao R Evaluating Protein Transfer Learning with TAPE. Adv. Neural Inf. Process. Syst 32, 9689– 9701 (2019). 33390682 PMC7774645 source
The Additional information section describes a sequence-length-invariant protein embedding used for remote homology detection and the same summary-embedding approach used to predict protein stability or fluorescence. Section 3 (Related Work) also describes applying NLP-based representation learning to biological sequences through transfer learning. Together, these support the claim about using learned protein representations to capture sequence relationships and connect sequence to function; no page numbers are available in our copy. (LLM, conf=0.88)
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [35] Hie B, Bryson BD & Berger B Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design. Cell Syst 11, 461– 477.e9 (2020). 33065027 source
In Additional information (page unavailable), the paper describes a pretrained neural language model that turns kinase sequences into features designed to encode structural and functional similarities. It also uses pretrained full-sequence embeddings to predict GFP fluorescence, supporting the claim that learned sequence representations can be transferred to a functional prediction task. (LLM, conf=0.88)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that our learned semantic embeddings dramatically outperform the sequence comparison methods and even outperform structure comparison with TMalign when predicting structural similarity.”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The knowledge distilled by these models allows us to improve downstream function prediction through transfer learning.”
- Also stated in: § Multi-task language models capture the semantic organization of proteins · boosted
“Here, we will demonstrate that training protein language models with self-supervision on a large amount of natural sequence data and with structure supervision on a smaller set of sequence, structure pairs enriches the learned representations and translates into improvements in downstream prediction problems (Figure 3).”
- Also stated in: § Transfer learning improves downstream applications · plain
“Learned protein representations improve predictive ability for downstream prediction problems through transfer learning (Figure 5a).”
§ Learning the Protein Language: Evolution, Structure and Function > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
- Rests on an assumption: We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
- Also stated in: § Multi-task language models capture the semantic organization of proteins · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that we can predict the phenotypes of unobserved sequence variants across datasets better than existing methods (Figure 5c).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
- Rests on an assumption: We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the BiLSTM+CRFs with our new embeddings (DLM-LSTM and MT-LSTM) outperform existing transmembrane predictors and a BiLSTM+CRF using our previous smaller embedding model (SSA-LSTM).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we observe that the structural supervision component is critical for learning well organized embeddings at a fine-grained level, because the DLM-LSTM representations alone do not perform well at this task (Supplemental Table 1).”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This parallels the general trend in natural language processing, where the number of parameters, rather than specific architectural choices, is the best indicator of model performance 52.”
§ Learning the Protein Language: Evolution, Structure and Function > Protein language models distill information from massive protein sequence databases
What the sources say
- Not checked yetcites [52] Kaplan J Scaling Laws for Neural Language Models. arXiv [cs.LG] (2020).
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Although structure is more informative of function than sequence 57,58 and structure is encoded by sequence, predicting structure remains hard, particularly due to the relative paucity of structural relative to sequence data.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
What the sources say
- Not checked yetcites [57] Zhang C & Kim S-H Overview of structural genomics: from structure to function. Curr. Opin. Chem. Biol 7, 28– 32 (2003). 12547423 source
- Not checked yetcites [58] Shin DH Structure-based inference of molecular functions of proteins of unknown function from Berkeley Structural Genomics Center. J. Struct. Funct. Genomics 8, 99– 105 (2007). 17764033 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Although it is generally believed that proteins with similar sequences form similar structures, there are also interesting examples of highly similar protein sequences having radically different structures and functions 60,61 and of sequences that can form multiple folds 62.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
What the sources say
- Not checked yetcites [60] Kosloff M & Kolodny R Sequence-similar, structure-dissimilar protein pairs in the PDB. Proteins 71, 891– 902 (2008). 18004789 PMC2673347 source
- Not checked yetcites [61] Wei KY Computational design of closely related proteins that adopt two well-defined but structurally divergent folds. Proc. Natl. Acad. Sci. U. S. A 117, 7208– 7215 (2020). 32188784 PMC7132107 source
- Not checked yetcites [62] James LC & Tawfik DS Conformational diversity and protein evolution--a 60-year-old hypothesis revisited. Trends Biochem. Sci 28, 361– 368 (2003). 12878003 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, unrelated protein sequences can form the same structural fold and, therefore, are semantically similar. However, we cannot deduce this relationship from sequences alone. Supervision is required to learn that these sequences belong to the same semantic category.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These phenomena suggest that there are aspects of protein biology that may not be discoverable by statistical sequence models alone. Supervision that represents known protein structure, function, and other prior knowledge may be necessary to encode distant sequence relationships into learned embeddings.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Proteins with similar structures and functions still exist below 30% sequence identity, but they can no longer be detected from sequence similarity alone and it is unclear whether their functions are conserved.
- Rests on an assumption: Evolutionary innovation requires that protein function can change with only a few mutations.
None of the sources cited for this claim have been checked yet.
In the paper
“Language models learn the probability of a sequence occurring and this can be directly applied to predict the fitness of sequence mutations 29-31 .”
§ Learning the Protein Language: Evolution, Structure and Function > Introduction
What the sources say
- Not checked yetcites [29] Riesselman AJ, Ingraham JB & Marks DS Deep generative models of genetic variation capture the effects of mutations. Nat. Methods 15, 816– 822 (2018). 30250057 PMC6693876 source
- Not checked yetcites [30] Hie B, Zhong E, Bryson B & Berger B Learning mutational semantics. Adv. Neural Inf. Process. Syst 33, (2020).
- Not checked yetcites [31] Hie B, Zhong ED, Berger B & Bryson B Learning the language of viral evolution and escape. Science 371, 284– 288 (2021). 33446556 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Language models capture complex relationships between residues in protein sequences by condensing information from enormous protein sequence databases.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fine-tuning pre-trained models can help with this problem but does not fully resolve it.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: For individual protein families or application-specific protein datasets, brute force hyperparameter search is the gold standard for selecting model architectures and number of parameters.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“High compute costs and murky design guidelines mean that developing new models is often an expensive, time consuming, and ad hoc process.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“To compare methods, we follow Luo et al and perform 5-fold cross validation on each deep mutational scanning dataset 34”
§ Learning the Protein Language: Evolution, Structure and Function > Methods > Sequence-to-phenotype prediction and evaluation
What the sources say
- Not matchedcites Luo et al
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We compare methods by 10-fold cross validation.”
§ Learning the Protein Language: Evolution, Structure and Function > Methods > Transmembrane region prediction training and evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“New linear complexity attention mechanisms may help to alleviate this limitation 95,96.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
What the sources say
- Not matchedcites [95] Choromanski KM Rethinking Attention with Performers. in International Conference on Learning Representations (2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [96] Wang S, Li BZ, Khabsa M, Fang H & Ma H Linformer: Self-Attention with Linear Complexity. (2020). source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Transformers scale quadratically with sequence length.
- Not matched
In the paper
“Recently, Luo et al. presented a method for combining language model-based representations with local evolutionary context-based representations (ECNet) and demonstrated that these representations were powerful for sequence-to-phenotype mapping on a panel of deep mutational scanning datasets 34.”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
What the sources say
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The paper relies on this without support.
Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
“A key challenge in biology is that many problems are small data problems. Quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.”
The paper relies on this without support.
We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
“Furthermore, we are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.”
The paper relies on this without support.
Proteins with similar structures and functions still exist below 30% sequence identity, but they can no longer be detected from sequence similarity alone and it is unclear whether their functions are conserved.
“Often called the “twilight zone” of protein sequence homology, proteins with similar structures and functions still exist below this level, but they can no longer be detected from sequence similarity alone and it is unclear whether their functions are conserved.”
The paper relies on this without support.
Evolutionary innovation requires that protein function can change with only a few mutations.
“Evolutionary innovation requires that protein function can change with only a few mutations.”
The paper relies on this without support.
Proteins exist in physical space.
“Proteins are objects that exist in physical space.”
The paper relies on this without support.
Many fundamental evolutionary processes that give rise to the diversity of protein sequences observed today are understood.
“Similarly, we understand many of the fundamental evolutionary processes that give rise to the diversity of protein sequences observed today.”
The paper relies on this without support.
Physics and evolution are key properties of proteins.
“These two elements, physics and evolution, are the key properties of proteins and our models might benefit from being structured explicitly to incorporate evolutionary and physics-based inductive biases.”
The paper relies on this without support.
Biological sequences are not natural language.
“Biological sequences are not natural language”
The paper relies on this without support.
Filtering sequences based on percent identity is a conventional crude address to the phylogenetic assumption problem.
“Conventionally, this problem is crudely addressed by filtering sequences based on percent identity.”
The paper relies on this without support.
At the upper limit of large natural protein databases, larger models continue to yield improved performance.
“At the upper limit of large natural protein databases, larger models continue to yield improved performance.”
The paper relies on this without support.
For individual protein families or application-specific protein datasets, brute force hyperparameter search is the gold standard for selecting model architectures and number of parameters.
“However, for individual protein families or other application specific protein data sets, the gold standard is to select model architectures and number of parameters via brute force hyperparameter search methods.”
The paper relies on this without support.
Transformers scale quadratically with sequence length.
“Transformers scale quadratically with sequence length”
The paper relies on this without support.
Recurrent neural networks scale linearly with sequence length.
“This problem is less extreme for recurrent neural networks, which scale linearly with sequence length”
The paper relies on this without support.
Increasing model size, compute power, and dataset size will continue to improve performance of protein language models.
“Increasing model size, compute power, and dataset size will only continue to improve performance of protein language models.”
The paper relies on this without support.
Protein language models are transforming computational protein biology today due to their ease of use and widespread applicability.
“Already, these methods are transforming computational protein biology today due to their ease of use and widespread applicability.”
The paper relies on this without support.
Augmenting language models with protein-specific properties such as structure and function offers an already successful route towards richer representations and novel biology.
“Furthermore, augmenting language models with protein specific properties such as structure and function offers one already successful route towards even richer representations and novel biology.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Instead of using multi-headed self-attention (MHSA), we adopt the Gated Attention Unit (GAU) (line 8 in Algorithm 1), which has shown great promise as an alternative to MHSAs with smaller memory consumption and faster convergence rate (25).”
From High-resolution de novo structure prediction from primary sequence (2022) · cites this paper as [25]
In the citing paper
“The original GAU in (25) also suggests that relu 2 (•) performs better compared to softmax in terms of both computation speed and convergence rate.”
From High-resolution de novo structure prediction from primary sequence (2022) · cites this paper as [25]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
MRBAYES: Bayesian inference of phylogenetic trees.
Huelsenbeck, J. P., & Ronquist, F. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.8.754 · Cited by 21,937
GROMACS 4: Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation
Hess, B., Kutzner, C., van der Spoel, D., & Lindahl, E. R. (2008) · Journal of Chemical Theory and Computation · doi:10.1021/ct700301q · Cited by 15,830
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759
The Pfam Protein Families Database
Bateman, A. G. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.276 · Cited by 14,075
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Needleman, S. B., & Wunsch, C. D. (1970) · Journal of Molecular Biology · doi:10.1016/0022-2836(70)90057-4 · Cited by 11,288
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
Comparison of multiple Amber force fields and development of improved protein backbone parameters
Horn̆ák, V., Abel, R., Okur, A., Strockbine, B., Roitberg, A. E., & Simmerling, C. L. (2006) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.21123 · Cited by 6,989
Accelerated Profile HMM Searches
Eddy, S. R. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002195 · Cited by 6,975
Gaussian Processes in Machine Learning
Rasmussen, C. E. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28650-9_4 · Cited by 6,522
HMMER web server: interactive sequence similarity searching
Finn, R. D., Clements, J., & Eddy, S. R. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr367 · Cited by 5,952
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
BEAST 2.5: An advanced software platform for Bayesian evolutionary analysis
Bouckaert, R., Vaughan, T. G., Barido-Sottani, J., Duchêne, S., Fourment, M., Gavryushkina, A., Heled, J., Jones, G., Kühnert, D., De Maio, N., et al. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006650 · Cited by 4,556
Pruitt, K. D., Tatusova, T. A., & Maglott, D. R. (2006) · Nucleic Acids Research · doi:10.1093/nar/gkl842 · Cited by 4,441
TM-align: a protein structure alignment algorithm based on the TM-score
Zhang, Y. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki524 · Cited by 3,774
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Generalizing from a Few Examples: A Survey on Few-shot Learning
Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2021) · ACM Computing Surveys · doi:10.1145/3386252 · Cited by 3,165
The Pfam protein families database
Bateman, A. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh121 · Cited by 3,127
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288
Leaver-Fay (2011) · doi:10.1016/b978-0-12-381270-4.00019-6 · Cited by 1,904
Twilight zone of protein sequence alignments
Rost, B. (1999) · Protein Engineering, Design and Selection · doi:10.1093/protein/12.2.85 · Cited by 1,833
Protein Structure Prediction Using Rosetta
Rohl, C. A., Strauss, C. E. M., Misura, K. M. S., & Baker, D. A. (2004) · Methods in enzymology on CD-ROM/Methods in enzymology · doi:10.1016/s0076-6879(04)83004-0 · Cited by 1,754
UniRef: comprehensive and non-redundant UniProt reference clusters
Suzek, B. E., Huang, H., McGarvey, P., Mazumder, R., & Wu, C. H. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm098 · Cited by 1,613
The Rosetta All-Atom Energy Function for Macromolecular Modeling and Design
Alford, R. F., Leaver-Fay, A., Jeliazkov, J. R., O’Meara, M. J., DiMaio, F. P., Park, H., Shapovalov, M. V., Renfrew, P. D., Mulligan, V. K., Kappel, K., et al. (2017) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.7b00125 · Cited by 1,532
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421
PhyloBayes 3: a Bayesian software package for phylogenetic reconstruction and molecular dating
Lartillot, N., Lepage, T., & Blanquart, S. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp368 · Cited by 1,297
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Deep Learning of Representations for Unsupervised and Transfer Learning.
Bengio, Y. (2011) · Cited by 1,080
SCOP: a Structural Classification of Proteins database
Hubbard, T., Murzin, A. G., Brenner, S. E., & Chothia, C. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.1.236 · Cited by 947
The TOPCONS web server for consensus prediction of membrane protein topology and signal peptides
Tsirigos, K. D., Peters, C., Shu, N., Käll, L., & Elofsson, A. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv485 · Cited by 939
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
Wang, S., Sun, S., Li, Z., Zhang, R., & Xu, J. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005324 · Cited by 931
Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition
Graves, A., Fernández, S., & Schmidhuber, J. (2005) · Lecture Notes in Computer Science · doi:10.1007/11550907_126 · Cited by 925
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867
Correlated mutations and residue contacts in proteins
Göbel, U., Sander, C., Schneider, R., & Valencia, A. (1994) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.340180402 · Cited by 863
Accurate classification of BRCA1 variants with saturation genome editing
Findlay, G. M., Daza, R. M., Martin, B., Zhang, M. D., Leith, A. P., Gasperini, M., Janizek, J. D., Huang, X., Starita, L. M., & Shendure, J. (2018) · Nature · doi:10.1038/s41586-018-0461-z · Cited by 843
GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU AccelerationPreprint
Gardner, J. R., Pleiss, G., Bindel, D., Weinberger, K. Q., & Wilson, A. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.11165 · Cited by 773
Generalizing from a Few Examples: A Survey on Few-Shot LearningPreprint
Wang, Y., Yao, Q., Kwok, J. T.-Y., & Ni, L. M.-S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05046 · Cited by 773
Fox, N. K., Brenner, S. E., & Chandonia, J.-M. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1240 · Cited by 762
Genealogical trees, coalescent theory and the analysis of genetic polymorphisms
Rosenberg, N. A., & Nordborg, M. (2002) · Nature Reviews Genetics · doi:10.1038/nrg795 · Cited by 750
MultiCoil: A program for predicting two‐and three‐stranded coiled coils
Wolf, E., Kim, P. S., & Berger, B. (1997) · Protein Science · doi:10.1002/pro.5560060606 · Cited by 720
Self-citationEvaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695
Iterated profile searches with PSI-BLAST—a tool for discovery in protein databases
Altschul, S. F., & Koonin, E. V. (1998) · Trends in Biochemical Sciences · doi:10.1016/s0968-0004(98)01298-5 · Cited by 679
Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., & Aurell, E. (2013) · Physical Review E · doi:10.1103/physreve.87.012707 · Cited by 661
Protein structure prediction from sequence variation
Marks, D. S., Hopf, T. A., & Sander, C. (2012) · Nature Biotechnology · doi:10.1038/nbt.2419 · Cited by 622
Predicting coiled coils by use of pairwise residue correlations.
Berger, B., Wilson, D. B., Wolf, E., Tonchev, T., Milla, M. E., & Kim, P. S. (1995) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.92.18.8259 · Cited by 622
Self-citationEmerging methods in protein co-evolution
de Juan, D., Pazos, F., & Valencia, A. (2013) · Nature Reviews Genetics · doi:10.1038/nrg3414 · Cited by 607
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601
Conformational diversity and protein evolution – a 60-year-old hypothesis revisited
James, L. C., & Tawfik, D. S. (2003) · Trends in Biochemical Sciences · doi:10.1016/s0968-0004(03)00135-x · Cited by 579
A Primer to Single-Particle Cryo-Electron Microscopy
Cheng, Y., Grigorieff, N., Penczek, P. A., & Walz, T. (2015) · Cell · doi:10.1016/j.cell.2015.03.050 · Cited by 570
Meta-Learning: A SurveyPreprint
Vanschoren, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03548 · Cited by 548
The Need for Biases in Learning Generalizations
Mitchell, T. M. (2007) · Cited by 514
CATH: increased structural coverage of functional space
Sillitoe, I., Bordin, N., Dawson, N., Waman, V. P., Ashford, P., Scholes, H. M., Pang, C. S. M., Woodridge, L., Rauer, C., Sen, N., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa1079 · Cited by 509
The spatial architecture of protein function and adaptation
McLaughlin, R. N., Poelwijk, F. J., Raman, A. S., Gosal, W. S., & Ranganathan, R. (2012) · Nature · doi:10.1038/nature11500 · Cited by 448
Multiplex assessment of protein variant abundance by massively parallel sequencing
Matreyek, K. A., Starita, L. M., Stephany, J. J., Martin, B., Chiasson, M. A., Gray, V. E., Kircher, M., Khechaduri, A., Dines, J. N., Hause, R. J., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0122-z · Cited by 436
Paircoil2: improved prediction of coiled coils from sequence.
McDonnell, A. V., Jiang, T., Keating, A. E., & Berger, B. (2006) · Bioinformatics · doi:10.1093/bioinformatics/bti797 · Cited by 424
Self-citationDistance-based protein folding powered by deep learning
Xu, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1821309116 · Cited by 401
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 383
End-to-End Differentiable Learning of Protein Structure
AlQuraishi, M. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.006 · Cited by 371
Elnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Fehér, T., Angerer, C. M., Steinegger, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06225 · Cited by 364
Learning the language of viral evolution and escape
Hie, B., Zhong, E. D., Berger, B., & Bryson, B. (2021) · Science · doi:10.1126/science.abd7331 · Cited by 361
Self-citationNavigating the protein fitness landscape with Gaussian processes
Romero, P. A., Krause, A., & Arnold, F. H. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1215251110 · Cited by 346
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
SARS-CoV-2 Variants of Concern in the United States—Challenges and Opportunities
Walensky, R. P., Walke, H. T., & Fauci, A. S. (2021) · JAMA · doi:10.1001/jama.2021.2294 · Cited by 319
Learned protein embeddings for machine learning
Yang, K., Wu, Z., Bedbrook, C. N., & Arnold, F. H. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty178 · Cited by 316
LINUS: a hierarchic procedure to predict the fold of a protein
Srinivasan, R., & Rose, G. D. (1995) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.340220202 · Cited by 266
Capturing the mutational landscape of the beta-lactamase TEM-1
Jacquier, H., Birgy, A., Nagard, H. L., Méchulam, Y., Schmitt, E., Glodt, J., Berçot, B., Petit, E., Poulain, J., Barnaud, G., et al. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1215206110 · Cited by 259
Generative models for graph-based protein design
Ingraham, J., Garg, V., Barzilay, R., & Jaakkola, T. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 258
Sledzieski, S., Singh, R., Cowen, L., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.08.010 · Cited by 235
Self-citationAraya, C. L., Fowler, D. M., Chen, W., Muniez, I., Kelly, J. W., & Fields, S. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1209751109 · Cited by 230
Deep mutational scanning of an RRM domain of the Saccharomyces cerevisiae poly(A)-binding protein
Melamed, D., Young, D. L., Gamble, C. E., Miller, C. R., & Fields, S. (2013) · RNA · doi:10.1261/rna.040709.113 · Cited by 223
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219
Self-citationA biologist’s guide to Bayesian phylogenetic analysis
Nascimento, F. F., Reis, M. D., & Yang, Z. (2017) · Nature Ecology & Evolution · doi:10.1038/s41559-017-0280-x · Cited by 212
A framework for exhaustively mapping functional missense variants
Weile, J., Sun, S., Cote, A. G., Knapp, J., Verby, M., Mellor, J. C., Wu, Y., Pons, C., Wong, C., van Lieshout, N., et al. (2017) · Molecular Systems Biology · doi:10.15252/msb.20177908 · Cited by 187
Activity-enhancing mutations in an E3 ubiquitin ligase identified by high-throughput mutagenesis
Starita, L. M., Pruneda, J. N., Lo, R. S., Fowler, D. M., Kim, H. J., Hiatt, J. B., Shendure, J., Brzovic, P. S., Fields, S., & Klevit, R. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1303309110 · Cited by 183
Phenotypic Characterization of a Comprehensive Set of MAPK1/ERK2 Missense Mutants
Brenan, L., Andreev, A., Cohen, O., Pantel, S., Kamburov, A., Cacchiarelli, D., Persky, N. S., Zhu, C., Bagul, M., Goetz, E. M., et al. (2016) · Cell Reports · doi:10.1016/j.celrep.2016.09.061 · Cited by 165
Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005
Attik, M., Bougrain, L., & Alexandre, F. (2005) · Lecture notes in computer science · doi:10.1007/11550907 · Cited by 164
Wrenbeck, E. E., Azouz, L. R., & Whitehead, T. A. (2017) · Nature Communications · doi:10.1038/ncomms15695 · Cited by 159
Antibody complementarity determining region design using high-capacity machine learning
Liu, G., Zeng, H., Mueller, J., Carter, B., Wang, Z., Schilz, J., Horny, G., Birnbaum, M. E., Ewert, S., & Gifford, D. K. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz895 · Cited by 157
Massively parallel single-amino-acid mutagenesis
Kitzman, J. O., Starita, L. M., Lo, R. S., Fields, S., & Shendure, J. (2015) · Nature Methods · doi:10.1038/nmeth.3223 · Cited by 154
Overview of structural genomics: from structure to function
Zhang, C., & Kim, S.-H. (2003) · Current Opinion in Chemical Biology · doi:10.1016/s1367-5931(02)00015-7 · Cited by 154
Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design
Hie, B., Bryson, B. D., & Berger, B. (2020) · Cell Systems · doi:10.1016/j.cels.2020.09.007 · Cited by 154
Self-citationRevolutionary cryo-EM is taking over structural biology
Callaway, E. (2020) · Nature · doi:10.1038/d41586-020-00341-9 · Cited by 153
Trade-offs between enzyme fitness and solubility illuminated by deep mutational scanning
Klesmith, J. R., Bacik, J. P., Wrenbeck, E. E., Michalczyk, R., & Whitehead, T. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1614437114 · Cited by 149
Analysis of distance‐based protein structure prediction by deep learning in CASP13
Xu, J., & Wang, S. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25810 · Cited by 132
Qian, B., Su, J., Wen, Z., Jha, D. N., Li, Y., Guan, Y., Puthal, D., James, P., Yang, R., Zomaya, A. Y., et al. (2021) · ACM Computing Surveys · doi:10.1145/3398020 · Cited by 132
Sequence‐similar, structure‐dissimilar protein pairs in the PDB
Kosloff, M., & Kolodny, R. (2008) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.21770 · Cited by 124
Deconstruction of the Ras switching cycle through saturation mutagenesis
Bandaru, P., Shah, N. H., Bhattacharyya, M., Barton, J. P., Kondo, Y., Cofsky, J. C., Gee, C. L., Chakraborty, A. K., Kortemme, T., Ranganathan, R., et al. (2017) · eLife · doi:10.7554/elife.27810 · Cited by 117
Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks
Liu, Y., Palmedo, P., Ye, Q., Berger, B., & Peng, J. (2018) · Cell Systems · doi:10.1016/j.cels.2017.11.014 · Cited by 114
Self-citationBedbrook, C. N., Yang, K., Rice, A. J., Gradinaru, V., & Arnold, F. H. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005786 · Cited by 113
Deciphering protein evolution and fitness landscapes with latent space models
Ding, X., Zou, Z., & Brooks III, C. L. (2019) · Nature Communications · doi:10.1038/s41467-019-13633-0 · Cited by 106
Bayesian Meta-Learning for the Few-Shot Setting via Deep KernelsPreprint
Patacchiola, M., Turner, J., Crowley, E. J., O’boyle, M. F. P., & Storkey, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05199 · Cited by 100
Chandonia, J., Fox, N. K., & Brenner, S. E. (2016) · Journal of Molecular Biology · doi:10.1016/j.jmb.2016.11.023 · Cited by 99
(2018) · International Conference on Learning Representations · Cited by 92
Trigg, J., Gutwin, K., Keating, A. E., & Berger, B. (2011) · PLoS One · doi:10.1371/journal.pone.0023519 · Cited by 91
Self-citationDe novo and inverse folding predictions of protein structure and dynamics
Godzik, A., Kolinski, A., & Skolnick, J. (1993) · Journal of Computer-Aided Molecular Design · doi:10.1007/bf02337559 · Cited by 91
Conditioning by adaptive sampling for robust designPreprint
Brookes, D. H., Park, H., & Listgarten, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10060 · Cited by 91
Zhou, G., Chen, M., Ju, C. J. ‐., Wang, Z., Jiang, J., & Wang, W. W. (2020) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqaa015 · Cited by 70
Zeng, H., & Gifford, D. K. (2019) · Cell Systems · doi:10.1016/j.cels.2019.05.004 · Cited by 67
Wei, K. Y., Moschidi, D., Bick, M. J., Nerli, S., McShan, A. C., Carter, L. P., Huang, P.-S., Fletcher, D. A., Sgourakis, N. G., Boyken, S. E., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914808117 · Cited by 57
Choi, J., & Pappu, R. V. (2019) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.8b00573 · Cited by 56
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42
Learned protein embeddings for machine learning
Yang, K., Wu, Z., Bedbrook, C. N., & Arnold, F. H. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty455 · Cited by 38
Algorithms for Protein Structural Motif Recognition
Berger, B. (1995) · Journal of Computational Biology · doi:10.1089/cmb.1995.2.125 · Cited by 31
Self-citationKim, S., Shin, D. H., Hou, J., Chandonia, J., Das, D., Choi, I., & Kim, R. (2007) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · Cited by 25
Research in Computational Molecular Biology
Chen, D., Jacob, L., & Mairal, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17083-7 · Cited by 15
Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering
Luo, Y., Vo, L. T., Ding, H., Su, Y., Liu, Y., Qian, W. W., Zhao, H., & Peng, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45257-5_30 · Cited by 15
Evolutionary context-integrated deep sequence modeling for protein engineeringPreprint
Luo, Y., Vo, L. T., Ding, H., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.01.16.908509 · Cited by 13
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
機械学習における不確実性の利用は生物学的発見と設計を加速する【JST・京大機械翻訳】
Brian, H., Bryan, D. B., & Bonnie, B. (2020) · Cell Systems · Cited by 9
Research in Computational Molecular Biology
(2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04749-7 · Cited by 7
(2020) · neural information processing systems · Cited by 3
Self-citationInterpretable deep learning framework for binding affinity prediction
Truong, T. F. (2020) · Cited by 2
Shin, D. H., Hou, J., Chandonia, J.-M., Das, D., Choi, I.-G., Kim, R., & Kim, S.-H. (2007) · Journal of Structural and Functional Genomics · doi:10.1007/s10969-007-9025-4
Research works printed but not linked: 4
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1
- Remmert, 2011, HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment, Nat. Methods, 9, 173, 10.1038/nmeth.1818 · DOI · Paper page
- Remmert, 2011, HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment, Nat. Methods, 9, 173, 10.1038/nmeth.1818 · DOI · Paper page
- Tsirigos, 2015, The TOPCONS web server for consensus prediction of membrane protein topology and signal peptides, Nucleic Acids Res., 43, W401-7, 10.1093/nar/gkv485 · DOI
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
“Supervised NLP methods, where the input sequences are trained jointly with their labels to produce predictive models, have been applied to various tasks, such as detecting structural similarity or predicting stability 3,4 .”
MethodEfficient evolution of human antibodies from general protein language models
Hie, B., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P. A. B., Tang, S., Wu, W. C. H., Pak, J. E., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 472
“1c) to learn patterns that are likely to occur in natural proteins 15–22 .”
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
“The OmegaFold model processes a protein sequence with a stack of GAU layers from ( 25 ) instaed of self-attention layers and multi-layer perceptrons.”
BackgroundMethodSelf-citation
Metadata
- Authors
- Bepler, T., & Berger, B.
- Affiliations
- New York Structural Biology Center (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Corresponding author
- Bepler, Tristan
- Berger, Bonnie
- Venue
- Cell Systems
- Publisher
- Elsevier BV
- Published
- 2021-06
- Volume
- 12
- Issue
- 6
- Pages
- 654-669.e3
- ISSN
- 2405-4712 (ISSN-L)
- PMC
- PMC8238390
- OpenAlex
- W3166142427
- Cited by
- 483
- References
- 133
- Citations by year
- 2026: 87 · 2025: 120 · 2024: 127 · 2023: 82 · 2022: 55 · 2021: 12
- FWCI
- 21.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Cell Systems (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-nd)
- Funders
- NIH (R35 GM141861)
- Simons Foundation (SF349247)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
- Keywords
- protein language models, protein sequence representation, protein structure prediction, protein function prediction, knowledge distillation, transfer learning, protein design, therapeutic design
- MeSH terms
- Machine Learning, Amino Acid Sequence, Language, Proteins, Databases, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
