The DOI matches its Crossref record; 130 of the 133 references it cites were found in a registry or an index; none of the 53 sources its claims cite is checked for support yet.
Learning the protein language: Evolution, structure, and function
Open accessAbstract
Language models have recently emerged as a powerful machine-learning approach for distilling information from massive protein sequence databases. From readily available sequence data alone, these models discover evolutionary, structural, and functional organization across protein space. Using language models, we can encode amino-acid sequences into distributed vector representations that capture their structural and functional properties, as well as evaluate the evolutionary fitness of sequence variants. We discuss recent advances in protein language modeling and their applications to downstream protein property prediction problems. We then consider how these models can be enriched with prior biological knowledge and introduce an approach for encoding protein structural knowledge into the learned representations. The knowledge distilled by these models allows us to improve downstream function prediction through transfer learning. Deep protein language models are revolutionizing protein biology. They suggest new ways to approach protein and therapeutic design. However, further developments are needed to encode strong biological priors into protein language models and to increase their accessibility to the broader community.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Protein language models learn evolutionary, structural, and functional organization from large sequence databases, and adding structure supervision improves downstream prediction through transfer learning. The authors present a multi-task LSTM trained on 76 million sequences plus 28,000 structures and show gains in structural similarity, transmembrane prediction, and sequence-to-phenotype prediction.
Key findings
- A multi-task LSTM (MT-LSTM) trained with masked language modeling, residue-residue contact prediction, and SCOP structural similarity prediction learned embeddings that “dramatically outperform” sequence comparison methods and even outperform TMalign for structural similarity; structure supervision was critical for fine-grained organization, and MT-LSTM outperformed the prior SSA-LSTM.
- For transmembrane region prediction, BiLSTM+CRF models using MT-LSTM embeddings significantly outperformed DLM-LSTM embeddings (paired t-test, p=0.044) and also outperformed existing predictors and SSA-LSTM.
- For sequence-to-phenotype prediction on deep mutational scanning datasets, MT-LSTM embeddings with Gaussian process regression achieved an average Spearman correlation of 0.65, significantly outperforming ECNet’s 0.60 (paired t-test, p=0.006).
- The paper notes that language models predicted SARS-CoV-2 spike mutations with high antibody-escape potential, but as of publication viral escape had not yet been experimentally verified.
Methods
- The authors trained a three-layered bidirectional LSTM with skip connections and 1024 hidden units in each direction per layer. The MT-LSTM combined three losses: masked language modeling (10% of amino acids randomly masked), residue-residue contact prediction (contacts defined as residues within 8 Å), and SCOP structural similarity prediction via ordinal regression; MT-LSTM loss weights were λmasked=0.5, λcontact=0.9, λsimilarity=0.1, while DLM-LSTM used only λmasked=1.
- Training data were 76,215,872 UniRef90 sequences (July 2018) and 28,010 SCOPe ASTRAL 2.06 protein sequences with structures, split into 22,408 training and 5,602 testing sequences. Models were trained for 1,000,000 parameter updates with Adam at learning rate 0.0001 on a single NVIDIA V100 GPU; training took roughly 13 days for DLM-LSTM and 51 days for MT-LSTM.
- Evaluations included structural similarity on 100,000 randomly sampled SCOPe ASTRAL test pairs, transmembrane prediction on the TOPCONS2 dataset of 5,154 proteins using 10-fold cross validation, and sequence-to-phenotype prediction on deep mutational scanning datasets using 5-fold cross validation and Gaussian process regression with an RBF kernel.
Limitations
- The paper states that further developments are needed to encode strong biological priors into protein language models and to increase their accessibility to the broader community; it also states that current models make heavily simplified phylogenetic assumptions, treating sequences as independent draws from a star phylogeny.
- The paper states practical limitations including high compute costs, murky design guidelines, ad hoc model development, quadratic scaling of transformers with sequence length (requiring long sequences to be excluded or truncated), and poor learning of long-range dependencies by RNNs.
- Evident from the text: the new results come from a single MT-LSTM architecture and specific benchmarks, and the SARS-CoV-2 escape predictions had not been experimentally verified as of publication.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 53 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also find that increasing model size improves transfer learning performance.”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this organization is improved when we include structure supervision in the language model training (Figure 4b).”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, representations learned by our MT-LSTM model significantly outperform (paired t-test, p=0.044) the embeddings learned by our DLM-LSTM model on this application (Figure 5b).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“Furthermore, the multi-task learning approach outperforms a two-step learning approach presented previously (SSA-LSTM)27.”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because we embed sequences into a semantic representation space, we can find structurally related proteins even though their sequences are not closely related (Figure 4c, Supplemental Table 1).”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that even though the DLM-LSTM model was trained using only sequence information, protein sequences still organize roughly by structure in embedding space.”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because TMscore is not symmetric, we calculate TMscore for both comparison directions and average them together for each protein pair. We found this outperformed other methods of combining the two scores.”
§ Learning the Protein Language: Evolution, Structure and Function > Methods > Protein structural similarity prediction evaluation
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIn the paper
“The key insight was that the sequence representations (vector embeddings) learned by the language model were powerful features for solving other prediction problems.”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“Interestingly, because so many sequences are available, these models continue to benefit from increased size 32.”
§ Learning the Protein Language: Evolution, Structure and Function > Protein language models distill information from massive protein sequence databases
What the sources say
- Not checked yetcites [32] Rives A Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv (2019). PMC8053943 33876751 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIn the paper
“Learned protein representations improve predictive ability for downstream prediction problems through transfer learning (Figure 5a).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
- Rests on an assumption: We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
- Probably not a substantial claim. 8
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that our learned semantic embeddings dramatically outperform the sequence comparison methods and even outperform structure comparison with TMalign when predicting structural similarity.”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The knowledge distilled by these models allows us to improve downstream function prediction through transfer learning.”
- Also stated in: § Multi-task language models capture the semantic organization of proteins · boosted
“Here, we will demonstrate that training protein language models with self-supervision on a large amount of natural sequence data and with structure supervision on a smaller set of sequence, structure pairs enriches the learned representations and translates into improvements in downstream prediction problems (Figure 3).”
- Also stated in: § Transfer learning improves downstream applications · plain
“Learned protein representations improve predictive ability for downstream prediction problems through transfer learning (Figure 5a).”
§ Learning the Protein Language: Evolution, Structure and Function > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
- Rests on an assumption: We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
- Also stated in: § Multi-task language models capture the semantic organization of proteins · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that we can predict the phenotypes of unobserved sequence variants across datasets better than existing methods (Figure 5c).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
- Rests on an assumption: We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the BiLSTM+CRFs with our new embeddings (DLM-LSTM and MT-LSTM) outperform existing transmembrane predictors and a BiLSTM+CRF using our previous smaller embedding model (SSA-LSTM).”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, we observe that the structural supervision component is critical for learning well organized embeddings at a fine-grained level, because the DLM-LSTM representations alone do not perform well at this task (Supplemental Table 1).”
§ Learning the Protein Language: Evolution, Structure and Function > Multi-task language models capture the semantic organization of proteins
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“This parallels the general trend in natural language processing, where the number of parameters, rather than specific architectural choices, is the best indicator of model performance 52.”
§ Learning the Protein Language: Evolution, Structure and Function > Protein language models distill information from massive protein sequence databases
What the sources say
- Not checked yetcites [52] Kaplan J Scaling Laws for Neural Language Models. arXiv [cs.LG] (2020).
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Although structure is more informative of function than sequence 57,58 and structure is encoded by sequence, predicting structure remains hard, particularly due to the relative paucity of structural relative to sequence data.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
What the sources say
- Not checked yetcites [57] Zhang C & Kim S-H Overview of structural genomics: from structure to function. Curr. Opin. Chem. Biol 7, 28– 32 (2003). 12547423 source
- Not checked yetcites [58] Shin DH Structure-based inference of molecular functions of proteins of unknown function from Berkeley Structural Genomics Center. J. Struct. Funct. Genomics 8, 99– 105 (2007). 17764033 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“They also learn summary representations, powerful features that can be used to better capture sequence relationships and link sequence to function via transfer learning 27,28,32-35.”
§ Learning the Protein Language: Evolution, Structure and Function > Introduction
What the sources say
- Not checked yetcites [27] Bepler T & Berger B Learning protein sequence embeddings using information from structure. in International Conference on Learning Representations (2019). PMC12947628 41768824 source
- Not checked yetcites [28] Alley EC, Khimulya G, Biswas S, AlQuraishi M & Church GM Unified rational protein engineering with sequence-based deep representation learning. Nat. Methods 16, 1315– 1322 (2019). 31636460 PMC7067682 source
- Not checked yetcites [32] Rives A Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv (2019). PMC8053943 33876751 source
- Not checked yetcites [33] Rao R Evaluating Protein Transfer Learning with TAPE. Adv. Neural Inf. Process. Syst 32, 9689– 9701 (2019). 33390682 PMC7774645 source
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [35] Hie B, Bryson BD & Berger B Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design. Cell Syst 11, 461– 477.e9 (2020). 33065027 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Although it is generally believed that proteins with similar sequences form similar structures, there are also interesting examples of highly similar protein sequences having radically different structures and functions 60,61 and of sequences that can form multiple folds 62.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
What the sources say
- Not checked yetcites [60] Kosloff M & Kolodny R Sequence-similar, structure-dissimilar protein pairs in the PDB. Proteins 71, 891– 902 (2008). 18004789 PMC2673347 source
- Not checked yetcites [61] Wei KY Computational design of closely related proteins that adopt two well-defined but structurally divergent folds. Proc. Natl. Acad. Sci. U. S. A 117, 7208– 7215 (2020). 32188784 PMC7132107 source
- Not checked yetcites [62] James LC & Tawfik DS Conformational diversity and protein evolution--a 60-year-old hypothesis revisited. Trends Biochem. Sci 28, 361– 368 (2003). 12878003 source
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, unrelated protein sequences can form the same structural fold and, therefore, are semantically similar. However, we cannot deduce this relationship from sequences alone. Supervision is required to learn that these sequences belong to the same semantic category.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These phenomena suggest that there are aspects of protein biology that may not be discoverable by statistical sequence models alone. Supervision that represents known protein structure, function, and other prior knowledge may be necessary to encode distant sequence relationships into learned embeddings.”
§ Learning the Protein Language: Evolution, Structure and Function > Supervision encodes biological meaning
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Proteins with similar structures and functions still exist below 30% sequence identity, but they can no longer be detected from sequence similarity alone and it is unclear whether their functions are conserved.
- Rests on an assumption: Evolutionary innovation requires that protein function can change with only a few mutations.
None of the sources cited for this claim have been checked yet.
In the paper
“Language models learn the probability of a sequence occurring and this can be directly applied to predict the fitness of sequence mutations 29-31 .”
§ Learning the Protein Language: Evolution, Structure and Function > Introduction
What the sources say
- Not checked yetcites [29] Riesselman AJ, Ingraham JB & Marks DS Deep generative models of genetic variation capture the effects of mutations. Nat. Methods 15, 816– 822 (2018). 30250057 PMC6693876 source
- Not checked yetcites [30] Hie B, Zhong E, Bryson B & Berger B Learning mutational semantics. Adv. Neural Inf. Process. Syst 33, (2020).
- Not checked yetcites [31] Hie B, Zhong ED, Berger B & Bryson B Learning the language of viral evolution and escape. Science 371, 284– 288 (2021). 33446556 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Language models capture complex relationships between residues in protein sequences by condensing information from enormous protein sequence databases.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fine-tuning pre-trained models can help with this problem but does not fully resolve it.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: For individual protein families or application-specific protein datasets, brute force hyperparameter search is the gold standard for selecting model architectures and number of parameters.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“High compute costs and murky design guidelines mean that developing new models is often an expensive, time consuming, and ad hoc process.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“To compare methods, we follow Luo et al and perform 5-fold cross validation on each deep mutational scanning dataset 34”
§ Learning the Protein Language: Evolution, Structure and Function > Methods > Sequence-to-phenotype prediction and evaluation
What the sources say
- Not matchedcites Luo et al
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We compare methods by 10-fold cross validation.”
§ Learning the Protein Language: Evolution, Structure and Function > Methods > Transmembrane region prediction training and evaluation
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“New linear complexity attention mechanisms may help to alleviate this limitation 95,96.”
§ Learning the Protein Language: Evolution, Structure and Function > Conclusions and Perspectives: Strong biological priors are key to improving protein language models
What the sources say
- Not matchedcites [95] Choromanski KM Rethinking Attention with Performers. in International Conference on Learning Representations (2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [96] Wang S, Li BZ, Khabsa M, Fang H & Ma H Linformer: Self-Attention with Linear Complexity. (2020). source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Transformers scale quadratically with sequence length.
- Not matched
In the paper
“Recently, Luo et al. presented a method for combining language model-based representations with local evolutionary context-based representations (ECNet) and demonstrated that these representations were powerful for sequence-to-phenotype mapping on a panel of deep mutational scanning datasets 34.”
§ Learning the Protein Language: Evolution, Structure and Function > Transfer learning improves downstream applications
What the sources say
- Not matchedcites [34] Luo Y Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering. in Research in Computational Molecular Biology 261– 263 (Springer International Publishing, 2020).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:boosted· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
The paper relies on this without support.
Many problems in biology are small data problems; quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.
“A key challenge in biology is that many problems are small data problems. Quantitative protein characterization assays are rarely high throughput and methods are needed that can generalize given only 10s to 100s of experimental measurements.”
The paper relies on this without support.
We are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.
“Furthermore, we are often interested in extrapolating from data collected over a small region of protein sequence space to other sequences, often with little to no homology.”
The paper relies on this without support.
Proteins with similar structures and functions still exist below 30% sequence identity, but they can no longer be detected from sequence similarity alone and it is unclear whether their functions are conserved.
“Often called the “twilight zone” of protein sequence homology, proteins with similar structures and functions still exist below this level, but they can no longer be detected from sequence similarity alone and it is unclear whether their functions are conserved.”
The paper relies on this without support.
Evolutionary innovation requires that protein function can change with only a few mutations.
“Evolutionary innovation requires that protein function can change with only a few mutations.”
The paper relies on this without support.
Proteins exist in physical space.
“Proteins are objects that exist in physical space.”
The paper relies on this without support.
Many fundamental evolutionary processes that give rise to the diversity of protein sequences observed today are understood.
“Similarly, we understand many of the fundamental evolutionary processes that give rise to the diversity of protein sequences observed today.”
The paper relies on this without support.
Physics and evolution are key properties of proteins.
“These two elements, physics and evolution, are the key properties of proteins and our models might benefit from being structured explicitly to incorporate evolutionary and physics-based inductive biases.”
The paper relies on this without support.
Biological sequences are not natural language.
“Biological sequences are not natural language”
The paper relies on this without support.
Filtering sequences based on percent identity is a conventional crude address to the phylogenetic assumption problem.
“Conventionally, this problem is crudely addressed by filtering sequences based on percent identity.”
The paper relies on this without support.
At the upper limit of large natural protein databases, larger models continue to yield improved performance.
“At the upper limit of large natural protein databases, larger models continue to yield improved performance.”
The paper relies on this without support.
For individual protein families or application-specific protein datasets, brute force hyperparameter search is the gold standard for selecting model architectures and number of parameters.
“However, for individual protein families or other application specific protein data sets, the gold standard is to select model architectures and number of parameters via brute force hyperparameter search methods.”
The paper relies on this without support.
Transformers scale quadratically with sequence length.
“Transformers scale quadratically with sequence length”
The paper relies on this without support.
Recurrent neural networks scale linearly with sequence length.
“This problem is less extreme for recurrent neural networks, which scale linearly with sequence length”
The paper relies on this without support.
Increasing model size, compute power, and dataset size will continue to improve performance of protein language models.
“Increasing model size, compute power, and dataset size will only continue to improve performance of protein language models.”
The paper relies on this without support.
Protein language models are transforming computational protein biology today due to their ease of use and widespread applicability.
“Already, these methods are transforming computational protein biology today due to their ease of use and widespread applicability.”
The paper relies on this without support.
Augmenting language models with protein-specific properties such as structure and function offers an already successful route towards richer representations and novel biology.
“Furthermore, augmenting language models with protein specific properties such as structure and function offers one already successful route towards even richer representations and novel biology.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Instead of using multi-headed self-attention (MHSA), we adopt the Gated Attention Unit (GAU) (line 8 in Algorithm 1), which has shown great promise as an alternative to MHSAs with smaller memory consumption and faster convergence rate (25).”
From High-resolution de novo structure prediction from primary sequence (2022) · cites this paper as [25]
In the citing paper
“The original GAU in (25) also suggests that relu 2 (•) performs better compared to softmax in terms of both computation speed and convergence rate.”
From High-resolution de novo structure prediction from primary sequence (2022) · cites this paper as [25]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
MRBAYES: Bayesian inference of phylogenetic trees.
Huelsenbeck, J. P., & Ronquist, F. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.8.754 · Cited by 21,937
GROMACS 4: Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation
Hess, B., Kutzner, C., van der Spoel, D., & Lindahl, E. R. (2008) · Journal of Chemical Theory and Computation · doi:10.1021/ct700301q · Cited by 15,830
Flagged first, then by citations
3 of the 133 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1
- Remmert, 2011, HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment, Nat. Methods, 9, 173, 10.1038/nmeth.1818 · DOI · Paper page
- Remmert, 2011, HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment, Nat. Methods, 9, 173, 10.1038/nmeth.1818 · DOI · Paper page
- Tsirigos, 2015, The TOPCONS web server for consensus prediction of membrane protein topology and signal peptides, Nucleic Acids Res., 43, W401-7, 10.1093/nar/gkv485 · DOI
Cited by
Sorted by citations
Items 201–300 of 486
Allo-Allo: Data-efficient prediction of allosteric sitesPreprint
Dong, T., Kan, C., Devkota, K., & Singh, R. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.28.615583 · Cited by 10
“To design an accurate data-efficient method for predicting allosteric sites, we leverage protein language models (PLMs) [2, 16, 8, 19, 12].”
MethodQiao, B., Wang, S., Hou, M., Chen, H., Zhou, Z., Xie, X., Pang, S., Yang, C., Yang, F., Zou, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae581 · Cited by 9
“We employed the pre-trained module to generate feature embeddings X N 2 R m × 6165 and X E 2 R n × 6165 for L N and L E , capturing both local and global aspects of the protein structure, function, and evolution across various dimensions (Bepler and Berger 2021).”
Alg-MFDL: A multi-feature deep learning framework for allergenic proteins prediction
Hu, X., Li, J., & Liu, T. (2024) · Analytical Biochemistry · doi:10.1016/j.ab.2024.115701 · Cited by 9
Supekar, S., Tay, D. W. P., Yeo, W. L., Tam, K. W. E., Koo, Y. S., See, J. Y., Miyajima, J. M. T., Maurer‐Stroh, S., Ang, E. L., Lim, Y. H., et al. (2024) · ACS Catalysis · doi:10.1021/acscatal.4c04660 · Cited by 9
A comprehensive benchmarking for evaluating TCR embeddings in modeling TCR-epitope interactions
Feng, X., Huo, M., Li, H., Yang, Y., Jiang, Y., He, L., & Li, S. C. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf030 · Cited by 10
Prediction of Adeno-Associated Virus Fitness with a Protein Language-Based Machine Learning Model
Wu, J., Qiu, Y., Lyashenko, E., Torregrosa, T., Pfister, E. L., Ryan, M. J., Mueller, C., & Choudhury, S. (2025) · Human Gene Therapy · doi:10.1089/hum.2024.227 · Cited by 10
“16 It is likely that different models encode different information and have their own inductive biases, 22 which are a function of which proteins are used for training and how they are represented in the model.”
Locality-aware pooling enhances protein language model performance across varied applications
Hoang, M., & Singh, M. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf178 · Cited by 10
“…models (PLMs) have recently emerged as a new technique for modeling the biological “grammar” that governs the behaviors of proteins (Alley et al. 2019, Bepler and Berger 2019, Heinzinger et al. 2019, Rao et al. 2019, Bepler and Berger 2021, Elnaggar et al. 2021, Rives et al. 2021, Lin et al. 2023).”
Dong, B., Liu, Z., Xu, D., Hou, C., Dong, G., Zhang, T., & Wang, G. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.03.018 · Cited by 13
“Proteins are fundamental components within living organisms, un-dertaking vital biological functions, such as signal transduction, information exchange, catalysis, and immune responses [1].”
BackgroundSEQUENCE VS. STRUCTURE: DELVING DEEP INTO DATA-DRIVEN PROTEIN FUNCTION PREDICTIONPreprint
Tian, X., Wang, Z., Yang, K., Su, J., Du, H., Zheng, Q., Guo, G., Yang, M., Yang, F., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.02.534383 · Cited by 8
“Researchers have adapted many natural language processing (NLP) methods to model AA sequences, known as protein language models (PLMs) (Bepler & Berger, 2021).”
Wang, Y., Wang, Z., Sadeh, G., Zancato, L., Achille, A., Karypis, G., & Rangwala, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.29.620988 · Cited by 12
“ProSE (Bepler & Berger, 2021) enhances these representations with structural supervision through residue-residue contact loss and structural similarity prediction, while CPCProt (Lu et al., 2020) leverages InfoNCE loss to maximize mutual information in protein embeddings.”
An Overview of Protein Function Prediction Methods: A Deep Learning Perspective
Ispano, E., Bianca, F., Lavezzo, E., & Toppo, S. (2023) · Current Bioinformatics · doi:10.2174/1574893618666230505103556 · Cited by 9
Zhang, J., Zhao, L., Wang, W. J., Zhang, Q., Wang, X., Xing, D., Ren, N., Lee, D., & Chen, C. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.172466 · Cited by 9
Frontiers in integrative structural modeling of macromolecular assemblies
Majila, K., Arvindekar, S. M., Jindal, M., & Viswanath, S. (2025) · QRB Discovery · doi:10.1017/qrd.2024.15 · Cited by 9
“…context-aware representations, providing a compact and informative approach to characterize the structural and functional properties of proteins (Bepler & Berger, 2021; Rives et al., 2021). pLMs enhance the performance of models on downstream tasks via transfer learning, eliminating the need to…”
Wang, Y., Zhu, X., & Wu, Z. (2025) · Digital Chemical Engineering · doi:10.1016/j.dche.2025.100231 · Cited by 8
SeaMoon: From protein language models to continuous structural heterogeneity
Lombard, V., Timsit, D., Grudinin, S., & Laine, E. (2025) · Structure · doi:10.1016/j.str.2025.06.010 · Cited by 8
Eskandar, K. (2025) · AI and Ethics · doi:10.1007/s43681-025-00872-9 · Cited by 11
“…models trained on hundreds of millions of sequences capture structure–function signals useful for zero-shot prediction of mutational effects (Bepler and Berger 2021), and scaled suc-cessors such as ESM-2 integrate information across billions of residues, producing state-of-the-art…”
The genetic foundations of convergent traits
Allard, J. B., & Kumar, S. (2026) · Nature Reviews Genetics · doi:10.1038/s41576-026-00933-7 · Cited by 13
“These embedding vectors can be extracted and used effectively as inputs for downstream prediction tasks 189–194 , which opens new avenues for linking genotypes to phenotypes.”
A flaw in using pretrained protein language models in protein–protein interaction inference models
Szymborski, J., & Emad, A. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-025-01176-7 · Cited by 13
Classifying alkaliphilic proteins using embeddings from protein language model
Susanty, M., Mursalim, M. K. N., Hertadi, R., Purwarianti, A., & Mengko, T. L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108385 · Cited by 10
Yang, Q., Jin, X., Zhou, H., Ying, J., Zou, J., Liao, Y., Lu, X., Ge, S., Yu, H. J., & Min, X. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108067 · Cited by 8
Improvements in viral gene annotation using large language models and soft alignments
Harrigan, W., Ferrell, B. D., Wommack, K. E., Polson, S. W., Schreiber, Z. D., & Belcaid, M. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05779-6 · Cited by 9
“…language-based approaches have been successfully applied to protein sequences revealing a continuous representation of amino acids in a high-dimensional space, where the amino acids show close proximity in vector space if they have similar functions in the context of the input sequence [19–21].”
MethodDeep-learning-based design of synthetic orthologs of SH3 signaling domains
Lian, X., Praljak, N., Subramanian, S., Wasinger, S., Ranganathan, R., & Ferguson, A. L. (2024) · Cell Systems · doi:10.1016/j.cels.2024.07.005 · Cited by 16
Trevizani, R., & Custódio, F. L. (2022) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2022.100038 · Cited by 8
Islam, M. N., Suzauddula, M., Ahamed, Z., Rabby, M. G., Hossen, M. M., Biswas, M., Bonny, M., & Hasan, M. M. (2022) · Archives of Microbiology · doi:10.1007/s00203-022-03270-5 · Cited by 8
“Threedimensional protein structure analysis aids in understanding the role of protein in different biological processes (Bepler and Berger 2021).”
BackgroundMethodInfluentialsolPredict: Antibody Apparent Solubility Prediction from Sequence by Transfer LearningPreprint
Feng, J., Jiang, M., Shih, J., & Chai, Q. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.12.07.471655 · Cited by 8
Comprehensive Overview of Bottom-up Proteomics using Mass SpectrometryPreprint
Jiang, Y., Rex, D. A. B., Schuster, D., Neely, B. A., Rosano, G. L., Volkmar, N., Momenzadeh, A., Peters-Clarke, T. M., Egbert, S., Kreimer, S. D., et al. (2023) · PubMed · doi:10.48550/arxiv.2311.07791 · Cited by 8
Protein language models uncover carbohydrate-active enzyme function in metagenomics
Thurimella, K., Mohamed, A. M. T., Li, C., Vatanen, T., Graham, D. B., Owens, R. M., La Rosa, S. L., Plichta, D. R., Bacallado, S., & Xavier, R. J. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06286-y · Cited by 9
“…models (pLMs), such as those used for structural prediction and other tasks, demonstrate remarkable capabilities in decoding the intricate amino acid language of proteins, which facilitates their functional annotation through a distinct approach compared to sequence-based alignment methods [34–38].”
Informatic challenges and advances in illuminating the druggable proteome
Taujale, R., Gravel, N., Zhou, Z., Yeung, W., Kochut, K., & Kannan, N. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.103894 · Cited by 7
Sun, S., Li, Q., Fan, L., Zhang, Y. ‐. P., & You, C. (2024) · ACS Sustainable Chemistry & Engineering · doi:10.1021/acssuschemeng.3c06089 · Cited by 7
Li, G., Zhang, N., & Fan, L. (2025) · ACS Omega · doi:10.1021/acsomega.4c09688 · Cited by 7
EvoNB: A protein language model-based workflow for nanobody mutation prediction and optimization
Xiong, D., Ming, Y., Li, Y., Li, S., Chen, K., Liu, J., Duan, L., Li, H., Li, M., & He, X. D. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101260 · Cited by 7
“Existing PLMs have demonstrated strong generalization capabilities across a wide range of protein sequence tasks.”
Zhang, N., Deng, Y., Li, R., Liu, F., Wei, D., Ren, Z., Jin, X., Zhao, T., Park, M., Choi, J., et al. (2025) · Free Radical Biology and Medicine · doi:10.1016/j.freeradbiomed.2025.08.015 · Cited by 7
Generative AI techniques for conformational diversity and evolutionary adaptation of proteins
Brownless, A.-L. R., Yehorova, D., Welsh, C. L., & Kamerlin, S. C. L. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103135 · Cited by 9
Kuang, J., Zhao, Z., Yang, Y., & Yan, W. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25158379 · Cited by 7
Wang, B., & Li, W. (2024) · Genes · doi:10.3390/genes15081090 · Cited by 7
“Language models (LMs) excel in content-aware data representation, from sequential databases, making them widely utilized in machine translation, question-answering systems, and even extended to applications in computer vision [54].”
BackgroundDeepFusionGO: Protein function prediction by fusing heterogeneous features through deep learning
Huang, Z., Zheng, R., & Deng, L. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9994899 · Cited by 7
“We use a pre-trained language model named Esm-1b [20]–[22] to obtain protein embedding that has been shown to achieve high-accuracy prediction results of protein contact by using simple logistic regression [22].”
MethodPrediction Of O-Glycosylation Site Using Pre-Trained Language Model And Machine Learning
Alkuhlani, A., Gad, W. K., Roushdy, M. I., & Salem, A.-B. M. (2023) · International journal of intelligent computing and information sciences/International Journal of Intelligent Computing and Information Sciences · doi:10.21608/ijicis.2023.160986.1218 · Cited by 7
Learning Complete Protein Representation by Deep Coupling of Sequence and StructurePreprint
Hu, B., Tan, C., Xia, J., Zheng, J., Huang, Y., Wu, L., Liu, Y., Xu, Y., & Li, S. Z. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.05.547769 · Cited by 7
“However, rich constraints between the sequence and structure of a protein, which may be critical for protein tasks [4], have yet to be fully explored.”
BackgroundProtein Language Models Uncover Carbohydrate-Active Enzyme Function in Metagenomics
Thurimella, K., Mohamed, A. M. T., Graham, D. B., Owens, R. M., La Rosa, S. L., Plichta, D. R., Bacallado, S., & Xavier, R. J. (2023) · doi:10.1101/2023.10.23.563620 · Cited by 7
“…(pLMs), such as those used for structural prediction and other tasks, demonstrate remarkable capabilities in decoding the intricate amino acid language of proteins, which facilitates their functional annotation through a distinct approach compared to sequence-based alignment methods 30,36–39 .”
BackgroundLanguage model-guided anticipation and discovery of unknown metabolitesPreprint
Qiang, H., Wang, F., Lu, W., Xing, X., Kim, H., Mérette, S. A. M., Ayres, L. B., Oler, E., AbuSalim, J. E., Roichman, A., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.13.623458 · Cited by 7
Wen, J., Ding, Z., Wei, Z., Xia, H., Zhang, Y., & Zhu, X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109048 · Cited by 6
HaloClass: Salt-Tolerant Protein Classification with Protein Language Models
Narang, K., Nath, A., Hemstrom, W., & Chu, S. K. S. (2024) · The Protein Journal · doi:10.1007/s10930-024-10236-7 · Cited by 6
“The recent development of protein language models (pLMs) has been an important advancement in computational and structural biology (9). pLMs, such as ESM-2, learn to predict the identity of masked amino acids using the context provided by the rest of the protein sequence (10).”
BackgroundLearning protein language contrastive models with multi-knowledge representation
Xu, W., Xia, Y., Sun, B., Zhao, Z., Tang, L., Zhou, X., Wang, Q., & Gu, L. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107580 · Cited by 6
Complex network structural analysis based on information supplementation graph contrastive learning
Cai, B., Wang, J., Tang, X., Li, X., Hu, N., Hu, Y., Liu, M., & Miao, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112833 · Cited by 6
Trans-MoRFs: A Disordered Protein Predictor Based on the Transformer Architecture
Meng, C., Shi, Y., Fu, X., Zou, Q., & Han, W. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3539710 · Cited by 7
“…years, significant advancements have been made in applying the transformer [45] architecture to protein large language modeling, and the explosion in the number of protein sequences allows us to use the transformer architecture to build protein language models (PLMs) [46], [47], [48], [49], [50].”
Evaluating sequence and structural similarity metrics for predicting shared paralog functions
Dennler, O., & Ryan, C. J. (2025) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqaf051 · Cited by 6
“It has been demonstrated that PLM embeddings efficiently capture key features of proteins, such as structural motifs, evolutionary signals, and functional elements [ 27 , 28 , 50 , 57 , 58 ].”
GeoEvoBuilder: A deep learning framework for efficient functional and thermostable protein design
Liu, J., You, H., Guo, Z., Xu, Q., Zhang, C., & Lai, L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2504117122 · Cited by 7
Learning the language of phylogeny with MSA Transformer
Chen, R., Foley, G., & Mikael Bodén, A. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101445 · Cited by 7
Zai, X., Zhao, Y., Wang, X., Leng, M., Lu, M., Yang, Y., Zhao, X., Li, R., Li, Y., Zhang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-67778-2 · Cited by 8
“Recent advancements in protein language models, such as ESM 23 , have shown promise in effectively representing and extracting feature information embedded within one-dimensional amino acid sequences 24 .”
Majila, K., Ullanat, V., & Viswanath, S. (2026) · Cell Systems · doi:10.1016/j.cels.2025.101486 · Cited by 8
Protein foundation models: a comprehensive survey
Xu, H., Li, L., Pan, S., Cheng, P., Wang, Y., Rong, Z., Feng, L., Huang, X., Wang, S., & Shu, W. (2026) · Science China Life Sciences · doi:10.1007/s11427-025-3147-2 · Cited by 8
A roadmap for exploring the untouched protein space for biology and medicine
Wang, J. (2023) · hLife · doi:10.1016/j.hlife.2023.06.001 · Cited by 6
“Among the various arti fi cial intelligence methods, natural language processing (NLP) approaches hold immense potential as they establish connections between protein functions (analogous to semantics in language processing) and sequences (akin to sentences) [34].”
Qu, Y., Niu, Z., Ding, Q., Zhao, T., Kong, T., Bai, B., Ma, J., Zhao, Y., & Zheng, J. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms242216496 · Cited by 7
“These studies are primarily rooted in the understanding that highly adaptive proteins are preserved during the extended process of natural protein evolution [4,20].”
BackgroundPLMC: Language Model of Protein Sequences Enhances Protein Crystallization Prediction
Xiong, D., Kaicheng, U., Sun, J., & Cribbs, A. P. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00639-6 · Cited by 6
“Keywords Protein crystallization · Protein language model · Deep learning · Multi-stage prediction · Globular and membrane proteins effectively capture complicated dependencies between amino acids [ 34 ].”
From Gene to Structure: Unraveling Genomic Dark Matter in Ca . Accumulibacter
Xie, X., Deng, X., Chen, L., Yuan, J., Chen, H., Wei, C., Feng, C., Liu, X., & Qiu, G. (2024) · Environmental Science & Technology · doi:10.1021/acs.est.4c09948 · Cited by 6
Elkin, M. E., & Xingquan Zhu, A. (2025) · Communications Biology · doi:10.1038/s42003-024-07262-7 · Cited by 6
“Natural Language Processing (NLP) methods have shown to tremendously bene fi t bioinformatics and protein research 34,35 , such as the AlphaFold 36 which predicts protein structures using sequences.”
Hui, W.-H., Chiu, P., Ng, I. I., Chang, S., Chou, C., & Chen, H.-H. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.02.017 · Cited by 7
“Every three amino acid sequences, GXY, in the α 1 chain and α 2 chain were transformed into 1024 vector embedding [56], as shown in Fig.”
MethodLearning the shape of protein micro-environments with a holographic convolutional neural networkPreprint
Pun, M. N., Ivanov, A., Bellamy, Q., Montague, Z., LaMont, C., Bradley, P. H., Otwinowski, J., & Nourmohammad, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.31.514614 · Cited by 7
Funk, J., Machado, L., Bradley, S. A., Napiorkowska, M., Gallegos-Dextre, R., Pashkova, L., Madsen, N. G., Webel, H. E., Phaneuf, P. V., Jenkins, T. P., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.01.616114 · Cited by 7
“[18– 21].”
Integrative modeling of seasonal influenza evolution via AI-powered antigenic cartographyPreprint
Ito, J., Kawakubo, S., Unno, H., Strange, A. P., Lytras, S., Okumura, K., Lilley, A., Harvey, R., Lewis, N. S., & Sato, K. (2025) · Research Square · doi:10.1101/2025.08.04.668423 · Cited by 9
“Notably, pLMs embed protein sequences into a semantic space that captures structural, functional, and evolutionary characteristics of the proteins 10,11 .”
Imam, I. A., Bailey, S., Wang, D., Zeng, S., Xu, D. H., & Shao, Q. (2024) · Langmuir · doi:10.1021/acs.langmuir.4c04140 · Cited by 5
An antibody developability triaging pipeline exploiting protein language models
Sweet-Jones, J., & Martin, A. C. R. (2025) · mAbs · doi:10.1080/19420862.2025.2472009 · Cited by 6
“43 These are deep learning encoders trained on the relationships between residues in a sequence using millions of sequences.”
Spicer, R., Raychawdhary, N., Danwada, S., Udomprasert, P., Seals, C., & Bhattacharya, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-23319-x · Cited by 5
“To overcome this, alignment-based strategies 37 – 39 are introduced, which compute residue-residue similarity using embedding-derived alignments or train neural networks to produce scoring and gap parameters.”
Generative AI for synthetic biology: Designing biological parts, circuits, and genomes
Kim, N., Carluccio, G. D., Zhang, K., & Collins, J. J. (2026) · Cell Systems · doi:10.1016/j.cels.2026.101533 · Cited by 5
“19,84,85 91 Representative examples include Meta’s Evolutionary Scale Modeling (ESM) series, 89–91 with ESM-3 supporting generative modeling through integrated sequence, structure, and function representations.”
Compressing the collective knowledge of ESM into a single protein language model
Dinh, T., Jang, S.-K., Zaitlen, N., & Ntranos, V. (2026) · Nature Methods · doi:10.1038/s41592-026-03050-9 · Cited by 7
Clark, J. D., Mi, X., Mitchell, D. A., & Shukla, D. (2024) · Digital Discovery · doi:10.1039/d4dd00170b · Cited by 8
Cai, F., Wei, Y., Kirchhofer, D., Chang, A. C., & Zhang, Y. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012609 · Cited by 5
Sequence modeling tools to decode the biosynthetic diversity of the human microbiome
Dason, M. S., Corà, D., & Re, A. (2025) · mSystems · doi:10.1128/msystems.00333-25 · Cited by 5
Zhang, Y., Qin, Y., Pourmirzaei, M., Shao, Q., Wang, D., & Xu, D. H. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_2 · Cited by 5
Progress and challenges for the application of machine learning for neglected tropical diseases
Khew, C., Akbar, R., & Mohd‐Assaad, N. (2023) · F1000Research · doi:10.12688/f1000research.129064.1 · Cited by 5
“…and treatment coverage data, which was at first unavailable probably due to absence of reports by the country ’ s Ministry of Health to the WHO, they were now able to access ample amounts of collected evidence pointing to the advantages of school-based deworming programs and LF MDA campaigns.”
Foundation models in molecular biology
Si, Y., Zou, J., Gao, Y., Chuai, G., Liu, Q., & Chen, L. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240006 · Cited by 6
“Language modeling has been very widely used in molecular biology after its great success in the field of natural language processing and has led to the research paradigm of “pre-training + fine-tuning” (Bepler and Berger 2021; Devlin et al .”
MethodAssessing the performance of protein regression modelsPreprint
Richard, M., Kæstel‐Hansen, J., Groth, P. M., Bartels, S., Salomon, J., Tian, P., Hatzakis, N. S., & Boomsma, W. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.18.545472 · Cited by 5
“Apart from a standard cross-validation approach (17;39;25;19;40;41;21), previous work includes different hold-outs (11), homology splitting (13), mutational or blocks of positions splitting (42;22), and task-inspired splitting (22;23).”
MethodMcWhite, C. D., & Singh, M. (2022) · doi:10.1101/2022.10.21.513099 · Cited by 4
Pseudo-perplexity in one fell swoop for protein fitness estimation
Kantroo, P., Wagner, G. P., & Machta, B. B. (2025) · PRX Life · doi:10.1103/zhx7-hcmm · Cited by 8
Prediction of enzyme function using an interpretable optimized ensemble learning framework
Dhibar, S., Basak, S., & Jana, B. (2025) · Chemical Science · doi:10.1039/d5sc04513d · Cited by 5
Mapping the space of protein binding sites with sequence-based protein language models
Oruç, T., Kadukova, M., Davies, T. G., Verdonk, M. L., & Poelking, C. R. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf284 · Cited by 4
“Given the embeddings for the whole sequence of length L (i.e., the L × 2560 matrix), a ligand-biased site embedding subsequently used as pocket representation was generated with mean pooling over the binding site residues.”
Reprogramming pretrained language models for protein sequence representation learning
Vinod, R., Chen, P., & Das, P. (2025) · Digital Discovery · doi:10.1039/d4dd00195h · Cited by 17
Zhu, X., Lu, J., Hu, X., Jin, T., Lu, S., & Feng, F. (2025) · Cell Reports · doi:10.1016/j.celrep.2025.115763 · Cited by 4
SOuLMuSiC, a novel tool for predicting the impact of mutations on protein solubility
Attanasio, S., Kwasigroch, J. M., Rooman, M. J., & Pucci, F. (2025) · Scientific Reports · doi:10.1038/s41598-025-11326-x · Cited by 6
“Including this model in our predictors allows us to leverage the evolutionary information embedded in the complex architecture of the pLM model 64 , without the need of performing a multiple sequence alignment for the target protein, thus preserving speed and scalability.”
UNICORN: Towards universal cellular expression prediction with a multi-task learning framework
Liu, T., Huang, T., Wang, L., Lin, Y., Ying, R., & Zhao, H. (2025) · Nature Communications · doi:10.1038/s41467-025-64506-8 · Cited by 5
GeNLP: a web tool for NLP-based exploration and prediction of microbial gene function
Miller, D., Arias, O., & Burstein, D. (2024) · Bioinform. · doi:10.1093/bioinformatics/btae034 · Cited by 3
“Language models have emerged as a valuable tool for various biological data analysis tasks, including the analysis of genomic and protein sequences (Bepler and Berger 2021, Lin et al. 2023, Madani et al. 2023).”
A CNN-CBAM-BIGRU model for protein function prediction
Sharma, L., Deepak, A., Ranjan, A., & Krishnasamy, G. (2024) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/sagmb-2024-0004 · Cited by 4
Vazzana, G., Savojardo, C., Martelli, P. L., & Casadio, R. (2024) · Molecules · doi:10.3390/molecules29194616 · Cited by 4
“…of Large Language Models (LLMs) for natural language processing tasks, transformer-based deep-learning architectures [1] have taken hold in the field of computational biology, with the consequent emergence of a counterpart adapted to protein sequences, known as protein Language Models (pLMs) [2,3].”
BackgroundMethodNguyen, V., Tran, T.-X., Nguyen, T.-T., & Le, N. Q. K. (2024) · Methods · doi:10.1016/j.ymeth.2024.10.006 · Cited by 4
Bai, T., Xie, J., Liu, Y., & Liu, B. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483997 · Cited by 5
“Inspired by the similarity between natural language and biological sequences [55], [56], [57], [58], our motivation is to enhance the feature representation by extracting diverse perspectives—lexical, syntactic, and semantic—form biological sequences.”
Zhao, X., Wang, Q., Zhang, Y., He, C., Yin, M., & Zhao, X. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3500027 · Cited by 4
LaGDif: Latent Graph Diffusion Model for Efficient Protein Inverse Folding with Self-Ensemble
Wu, T. T., Wang, Y. G., & Shen, Y. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821758 · Cited by 4
“These language-based models typically fall into two categories, namely the masked models and autoregressive models [8].”
Contrastive learning enables epitope overlap predictions for targeted antibody discovery
Holt, C., Janke, A. K., Amlashi, P., Jamieson, P. J., Marinov, T. M., & Georgiev, I. S. (2025) · Patterns · doi:10.1016/j.patter.2025.101419 · Cited by 5
Protein sequence analysis in the context of drug repurposing
Sánchez, N. G., Carro, E. U., Santamaría, L. P., & Rodríguez‐González, A. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02531-1 · Cited by 5
“Furthermore, the emergence of large-scale data has generated more interest in efficient data-driven approaches rather than high-precision property-based experimental data methods that are more time-consuming [16].”
BackgroundMethodPokharel, S., Pratyush, P., Heinzinger, M., Newman, R. H., & KC, D. B. (2022) · Research Square · doi:10.21203/rs.3.rs-1953874/v1 · Cited by 4
Predicting enzyme substrate chemical structure with protein language modelsPreprint
Jinich, A., Nazia, S. Z., Tellez, A. V., Rappoport, D., AlQuraishi, M., & Rhee, K. Y. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.09.28.509940 · Cited by 4
“Particularly exciting are protein language models, a family of machine learning techniques adapted from the field of natural language processing (NLP) and tailored for protein sequence analysis 19–21 .”
BackgroundArtificial Intelligence Learns Protein PredictionPreprint
Heinzinger, M., & Rost, B. (2024) · Cold Spring Harbor Perspectives in Biology · doi:10.1101/cshperspect.a041458 · Cited by 4
Interpretable machine learning reveals a diverse arsenal of anti-defenses in bacterial virusesPreprint
Lopatina, A., Ferenc, M., Bartlau, N., Wolfram, M., Steensen, K., Muscia, S., Hussain, F. A., Schnurer, M., Afjehi‐Sadat, L., Pollak, S., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.14.598830 · Cited by 7
ProteinAligner: A Multi-modal Pretraining Framework for Protein Foundation ModelsPreprint
Zhang, L., Guo, H., Schaffer, L. V., Ko, Y. S., Singh, D., Rahmani, H., Grotjahn, D., Villa, E., Gilson, M., Wang, W., et al. (2024) · bioRxiv · doi:10.1101/2024.10.06.616870 · Cited by 6
FLIP2: Expanding Protein Fitness Landscape Benchmarks for Real-World Machine Learning ApplicationsPreprint
Didi, K., Alamdari, S., Lu, A. X., Wittmann, B. J., Johnston, K. E., Amini, A. P., Madani, A., Czeneszew, M., Dallago, C., & Yang, K. K. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.23.707496 · Cited by 7
“More recently, modern pLMs were shown to learn evolutionary and structural information by training on unlabeled sequences and structures (Heinzinger et al., 2019; Rao et al., 2019; Bepler & Berger, 2021; Rives et al., 2021; Zhang et al., 2024).”
Advances in Protein Function Prediction from the Fifth CAFA ChallengePreprint
Kaluza, M. C. D. P., Ramola, R., Joshi, P., Piovesan, D., Reade, W., Orchard, S. E., Martin, M. J., Ignatchenko, A., Rost, B., Orengo, C. A., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.27.716980 · Cited by 5
“For the similarity measure, we used cosine and Euclidean distances between per-protein embeddings which has been shown to capture relevant protein relationships [7, 16, 33, 24].”
Piskova, T., Kozyrina, A. N., Astrauskaite, G., Mabrouk, M. H. E., Schepl, S., Yam, S. L. S., Ravithas, R., Wagner, W., Vassalli, M., & Russo, J. D. (2026) · Nature Communications · doi:10.1038/s41467-026-71493-x · Cited by 4
“This paradigm applies across various length scales, from the molecular 3,4 and sub-cellular levels 5,6 to the tissue and organ levels, where proper architecture and cellular connectivity are essential for functionality 7 – 9 .”
Phage evolutionary relationships emerge from protein language model-based proteome representation
Panigrahi, S., Ansaldi, M., & Ginet, N. (2025) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqaf134 · Cited by 3
“…this study builds on the advancements seen in pLMs and the ESM Metagenomic Atlas published only 2 years ago [ 7 ]. pLMs have shown an impressive ability to learn the ‘language’ of proteins in a self-supervised manner, providing insights into their functions and evolutionary relationships [ 35 ].”
ProGraphTrans: multimodal dynamic collaborative framework for protein representation learning
Li, Z., Liu, Y., Han, G., Yu, Z.-G., & Liu, Y. (2025) · BMC Biology · doi:10.1186/s12915-025-02432-3 · Cited by 3
“The current research paradigm usually follows the path from representation method development to task application validation, in which researchers first propose novel protein representation methods, and then solve the downstream tasks through direct application or adaptation [3, 4].”
Metadata
- Authors
- Bepler, T., & Berger, B.
- Affiliations
- New York Structural Biology Center (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Corresponding author
- Bepler, Tristan
- Berger, Bonnie
- Venue
- Cell Systems
- Publisher
- Elsevier BV
- Published
- 2021-06
- Volume
- 12
- Issue
- 6
- Pages
- 654-669.e3
- ISSN
- 2405-4712 (ISSN-L)
- PMC
- PMC8238390
- OpenAlex
- W3166142427
- Cited by
- 483
- References
- 133
- Citations by year
- 2026: 87 · 2025: 120 · 2024: 127 · 2023: 82 · 2022: 55 · 2021: 12
- FWCI
- 21.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Cell Systems (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-nd)
- Funders
- NIH (R35 GM141861)
- Simons Foundation (SF349247)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
- Keywords
- protein language models, protein sequence representation, protein structure prediction, protein function prediction, knowledge distillation, transfer learning, protein design, therapeutic design
- MeSH terms
- Machine Learning, Amino Acid Sequence, Language, Proteins, Databases, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
