Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Rives, Alexander; Meier, Joshua; Sercu, Tom; Goyal, Siddharth; Lin, Zeming; Liu, Jason; Guo, Demi; Ott, Myle; Zitnick, C. Lawrence; Ma, Jerry; Fergus, Rob

doi:10.1101/622803

PreprintCited by 3,280Machine Learning in BioinformaticsProtein Structure and DynamicsRNA and protein synthesis mechanisms

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Open access

Rives, Alexander, Meier, Joshua, Sercu, Tom, Goyal, Siddharth, Lin, Zeming, Liu, Jason, Guo, Demi, Ott, Myle +3 more·2019·Proceedings of the National Academy of Sciences

DOI 10.1101/622803PMC PMC8053943

Preprint — published as 10.1073/pnas.2016239118 · Paper page

Open this paper in CoChat — free account

Abstract

In the field of artificial intelligence, a combination of scale in data and model capacity enabled by un-supervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multi-scale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure, and improving state-of-the-art features for long-range contact prediction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1101/622803

Cites

148 of 148 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3× · see citing works
“We investigate the Transformer (Vaswani et al., 2017), which has emerged as a powerful general-purpose model architecture for representation learning and generative modeling of textual data, and outperforms more traditionally employed recurrent and convolutional architectures, such as LSTM and GConv networks (Hochreit…”
Method
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 5× · see citing works
“…a form of unsupervised learning in which context within the text is used to predict missing words, have been shown to materialize representations of word meaning that can generalize across problems (Collobert & Weston, 2008; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2018).”
BackgroundMethodInfluential
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1× · see citing works
“A.1 B ACKGROUND ON LANGUAGE MODELS AND EMBEDDINGS Deep language models are parametric functions that map sequences into distributed word representations in a learned vector space (Bengio et al., 2003; Mikolov et al., 2013a;b; Pennington et al., 2014; Lample et al., 2017).”
Background
Untitled work
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1× · see citing works
“A.1 B ACKGROUND ON LANGUAGE MODELS AND EMBEDDINGS Deep language models are parametric functions that map sequences into distributed word representations in a learned vector space (Bengio et al., 2003; Mikolov et al., 2013a;b; Pennington et al., 2014; Lample et al., 2017).”
Background
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1× · see citing works
“A.1 B ACKGROUND ON LANGUAGE MODELS AND EMBEDDINGS Deep language models are parametric functions that map sequences into distributed word representations in a learned vector space (Bengio et al., 2003; Mikolov et al., 2013a;b; Pennington et al., 2014; Lample et al., 2017).”
Background
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Dictionary of protein secondary structure: Pattern recognition of hydrogen‐bonded and geometrical features
Kabsch, W., & Sander, C. (1983) · Biopolymers · doi:10.1002/bip.360221211 · Cited by 15,320 · see citing works
Untitled work
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2× · see citing works
“Peters et al. (2018) used a coupled language model objective to train bidirectional LSTMs in order to extract context dependent word embeddings.”
BackgroundMethod
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
Analysis of protein-coding genetic variation in 60,706 humans
Exome Aggregation Consortium, Lek, M., Karczewski, K. J., Minikel, E. V., Samocha, K. E., Banks, E., Fennell, T., O’Donnell-Luria, A. H., Ware, J. S., Hill, A. J., et al. (2016) · Nature · doi:10.1038/nature19057 · Cited by 9,587 · cited 1× · see citing works
“The mutational fitness landscape provides deep insight into biology: determining protein activity (Fowler & Fields, 2014), linking genetic variation to human health (Lek et al., 2016; Bycroft et al., 2018; Karczewski et al., 2019), and enabling for rational protein engineering (Slaymaker et al.”
Background
The UK Biobank resource with deep phenotyping and genomic data
Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L. T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O’Connell, J., et al. (2018) · Nature · doi:10.1038/s41586-018-0579-z · Cited by 8,876 · cited 1× · see citing works
“The mutational fitness landscape provides deep insight into biology: determining protein activity (Fowler & Fields, 2014), linking genetic variation to human health (Lek et al., 2016; Bycroft et al., 2018; Karczewski et al., 2019), and enabling for rational protein engineering (Slaymaker et al.”
Background
Predicting the effects of coding non-synonymous variants on protein function using the SIFT algorithm
Kumar, P., Henikoff, S., & Ng, P. C. (2009) · Nature Protocols · doi:10.1038/nprot.2009.86 · Cited by 6,521 · cited 1× · see citing works
“Computational variant effect predictors are useful for assessing the effect of point mutations (Gray et al., 2018; Adzhubei et al., 2013; Kumar et al., 2009; Hecht et al., 2015; Rentzsch et al., 2018).”
Background
Pfam: the protein families database
Finn, R. D., Bateman, A., Clements, J., Coggill, P., Eberhardt, R. Y., Eddy, S. R., Heger, A., Hetherington, K., Holm, L., Mistry, J., et al. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1223 · Cited by 6,148 · see citing works
Protein secondary structure prediction based on position-specific scoring matrices.
Jones, D. C. (1999) · Journal of Molecular Biology · doi:10.1006/jmbi.1999.3091 · Cited by 5,741 · cited 2× · see citing works
“…we compare to (1) one-hot amino acid representations, which results in the prior that predicts the most likely label for each position’s amino acid, and (2) representations from family-level frequencies summarized in position-speciﬁc scoring matrices (“PSSM”s; Jones, 1999) for each protein.”
Method
Profile hidden Markov models
Eddy, S. R. (1998) · Bioinformatics · doi:10.1093/bioinformatics/14.9.755 · Cited by 5,515 · cited 1× · see citing works
“Efficient sequence search methods have been the computational foundation for extracting biological information from sequences (Altschul et al., 1990; Altschul & Koonin, 1998; Eddy, 1998; Remmert et al., 2011).”
Method
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 1× · see citing works
“…a form of unsupervised learning in which context within the text is used to predict missing words, have been shown to materialize representations of word meaning that can generalize across problems (Collobert & Weston, 2008; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2018).”
Background
eggNOG 5.0: a hierarchical, functionally and phylogenetically annotated orthology resource based on 5090 organisms and 2502 viruses
Huerta‐Cepas, J., Szklarczyk, D., Heller, D., Hernández-Plaza, A., Forslund, S. K., Cook, H., Mende, D. R., Letunić, I., Rattei, T., Jensen, L. J., et al. (2018) · Nucleic Acids Research · doi:10.1093/nar/gky1085 · Cited by 4,943 · cited 2× · see citing works
“Generally, these genes will have different sequences but retain the same function (Huerta-Cepas et al., 2018).”
BackgroundMethodInfluential
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · see citing works
Distributional Structure
Harris, Z. S. (1954) · <i>WORD</i> · doi:10.1080/00437956.1954.11659520 · Cited by 4,807 · see citing works
Semi-Supervised Learning
Chapelle, O., Schölkopf, B., & Zien, A. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262033589.001.0001 · Cited by 4,560 · cited 1× · see citing works
“For the analysis in Section 4.3, we deﬁne various translation operations on the representation space of the diverse orthology dataset described above.”
MethodInfluential
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
The Universal Protein Resource (UniProt)
Bairoch, A. (2004) · Nucleic Acids Research · doi:10.1093/nar/gki070 · Cited by 3,774 · see citing works
Predicting Functional Effect of Human Missense Mutations Using PolyPhen‐2
Adzhubei, I., Jordan, D. M., & Sunyaev, S. R. (2013) · Current Protocols in Human Genetics · doi:10.1002/0471142905.hg0720s76 · Cited by 3,705 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · cited 1× · see citing works
“All models were trained using the fairseq toolkit (Ott et al., 2019) on 128 NVIDIA V100 GPUs for ˜ 4 days with sinusoidal positional embeddings (Vaswani et al., 2017), without layer norm in the ﬁnal layer, and using half A.6 O RTHOLOGY VISUALATIONS Full orthology dataset For the analyses in Section…”
MethodSelf-citation
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
CADD: predicting the deleteriousness of variants throughout the human genome
Rentzsch, P., Witten, D., Cooper, G. M., Shendure, J., & Kircher, M. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1016 · Cited by 3,534 · cited 1× · see citing works
“Computational variant effect predictors are useful for assessing the effect of point mutations (Gray et al., 2018; Adzhubei et al., 2013; Kumar et al., 2009; Hecht et al., 2015; Rentzsch et al., 2018).”
Background
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451 · see citing works
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378 · cited 2× · see citing works
“State-of-the-art neural networks for tertiary structure and contact prediction use deep residual architectures with two-dimensional convolutions over pairwise feature maps to output a contact prediction or distance potential for each pair of residues (54, 56, 59).”
Background
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Self-citation
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,646 · see citing works
CATH – a hierarchic classification of protein domain structures
Orengo, C., Michie, A. D., Jones, S., Jones, D., Swindells, M. B., & Thornton, J. M. (1997) · Structure · doi:10.1016/s0969-2126(97)00260-8 · Cited by 2,608 · see citing works
Rationally engineered Cas9 nucleases with improved specificity
Slaymaker, I. M., Gao, L., Zetsche, B., Scott, D. A., Yan, W. X., & Zhang, F. (2016) · Science · doi:10.1126/science.aad5227 · Cited by 2,414 · cited 1× · see citing works
“…mutational ﬁtness landscape provides deep insight into biology: determining protein activity (Fowler & Fields, 2014), linking genetic variation to human health (Lek et al., 2016; Bycroft et al., 2018; Karczewski et al., 2019), and enabling for rational protein engineering (Slaymaker et al., 2016).”
Background
Protein homology detection by HMM–HMM comparison
Söding, J. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bti125 · Cited by 2,381 · cited 1× · see citing works
“…the model to predict the effect of changing a single amino acid (e.g. for mutagenesis prediction), we regress l π ( mtAA ) l π ( wtAA ) to the scaled mutational effect where π maps an amino acid to its corresponding index; mt AA is the mutant amino acid; and wt AA is the wildtype amino acid.”
Background
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288 · cited 1× · see citing works
“Efficient sequence search methods have been the computational foundation for extracting biological information from sequences (Altschul et al., 1990; Altschul & Koonin, 1998; Eddy, 1998; Remmert et al., 2011).”
Method
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 1× · see citing works
“Randomly initialized networks can produce features that perform well without any learning (41).”
Background
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928 · cited 2× · see citing works
“We use UniRef (Suzek et al., 2015) to create three pre-training datasets with differing levels of diversity: (i) the low-diversity dataset (UR100) uses the UniRef100 representative sequences; (ii) the high-diversity sparse dataset (UR50/S) uses the UniRef50 representative sequences; (iii) the high-diversity dense data…”
Method
The mutational constraint spectrum quantified from variation in 141,456 humans
Karczewski, K. J., Francioli, L. C., Tiao, G., Cummings, B. B., Alföldi, J., Wang, Q., Collins, R. L., Laricchia, K. M., Ganna, A., Birnbaum, D. P., et al. (2019) · doi:10.1101/531210 · Cited by 1,662 · cited 1× · see citing works
“…mutational ﬁtness landscape provides deep insight into biology: determining protein activity (Fowler & Fields, 2014), linking genetic variation to human health (Lek et al., 2016; Bycroft et al., 2018; Karczewski et al., 2019), and enabling for rational protein engineering (Slaymaker et al., 2016).”
Background
UniRef: comprehensive and non-redundant UniProt reference clusters
Suzek, B. E., Huang, H., McGarvey, P., Mazumder, R., & Wu, C. H. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm098 · Cited by 1,613 · see citing works
Evolved Cas9 variants with broad PAM compatibility and high DNA specificity
Hu, J. H., Miller, S. M., Geurts, M. H., Tang, W., Chen, L., Sun, N., Zeina, C. M., Gao, X., Rees, H. A., Lin, Z., et al. (2018) · Nature · doi:10.1038/nature26155 · Cited by 1,552 · cited 1× · see citing works
“…the introduction of mutations to enhance protein function (Romero & Arnold, 2009), inform protein design using a combination of activating mutants (Hu et al., 2018), and make rational substitutions to optimize protein properties such as substrate speciﬁcity (Packer et al., 2017), stability (Tan…”
BackgroundSelf-citation
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D. A., & Rush, A. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,546 · see citing works
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530 · see citing works
Direct-coupling analysis of residue coevolution captures native contacts across many protein families
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434 · cited 1× · see citing works
“Maximum entropy methods disentangle direct interactions from indirect interactions by inferring parameters of a posited generating distribution for the sequence family (Weigt et al., 2009; Marks et al., 2011; Morcos et al., 2011; Jones et al., 2011; Balakrishnan et al., 2011; Ekeberg et al., 2013b).”
Method
Multiobjective evolutionary algorithms to identify highly autocorrelated areas: the case of spatial distribution in financially compromised farms
GARCÍA-ALONSO, C. R., Pérez, L. M., & Fernández, J. C. F. (2011) · Annals of Operations Research · doi:10.1007/s10479-011-0841-3 · Cited by 1,357 · see citing works
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312 · see citing works
Deep mutational scanning: a new style of protein science
Fowler, D. M., & Fields, S. (2014) · Nature Methods · doi:10.1038/nmeth.3027 · Cited by 1,287 · see citing works
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Exploring protein fitness landscapes by directed evolution
Romero, P. A., & Arnold, F. H. (2009) · Nature Reviews Molecular Cell Biology · doi:10.1038/nrm2805 · Cited by 1,163 · cited 2× · see citing works
“…iteratively querying a model of the mutational ﬁtness landscape could help efﬁciently guide the introduction of mutations to enhance protein function (Romero & Arnold, 2009), inform protein design using a combination of activating mutants (Hu et al., 2018), and make rational substitutions to…”
Background
HH-suite3 for fast remote homology detection and deep protein annotation
Steinegger, M., Meier, M., Mirdita, M., Vöhringer, H., Haunsberger, S. J., & Söding, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3019-7 · Cited by 1,145 · see citing works
Protein 3D Structure Computed from Evolutionary Sequence Variation
Marks, D. S., Colwell, L. J., Sheridan, R., Hopf, T. A., Pagnani, A., Zecchina, R., & Sander, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028766 · Cited by 1,144 · cited 1× · see citing works
“Contacts can be inferred through evolutionary correlations between residues in a sequence family (Marks et al., 2011; Anishchenko et al., 2017).”
Background
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · see citing works
Machine-learning-guided directed evolution for protein engineering
Yang, K., Wu, Z., & Arnold, F. H. (2019) · Nature Methods · doi:10.1038/s41592-019-0496-6 · Cited by 1,082 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · cited 1× · see citing works
“Layer norm placement Recent works Child et al. (2019) and Shoeybi et al. (2020) have suggested that pre-activation layer norm results in more stable training for larger models.”
Background
A series of PDB related databases for everyday needs
Joosten, R., Beek, T. T. T., Krieger, E., Hekkelman, M., Hooft, R., Schneider, R., Sander, C., & Vriend, G. (2010) · Nucleic Acids Res. · doi:10.1093/nar/gkq1105 · Cited by 1,050 · see citing works
Identification of direct residue contacts in protein–protein interaction by message passing
Weigt, M., White, R. A., Szurmant, H., Hoch, J. A., & Hwa, T. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0805923106 · Cited by 1,025 · cited 2× · see citing works
“Raw covariance, correlation, and mutual information have confounding effects from indirect couplings (Weigt et al., 2009).”
BackgroundMethod
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 1× · see citing works
“…a form of unsupervised learning in which context within the text is used to predict missing words, have been shown to materialize representations of word meaning that can generalize across problems (Collobert & Weston, 2008; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2018).”
Background
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
Wang, S., Sun, S., Li, Z., Zhang, R., & Xu, J. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005324 · Cited by 931 · cited 15× · see citing works
“This z 1 ∈ R 2 d 2 × L × L is then concatenated in the ﬁrst (feature map or channel) dimension, with the pairwise features z 2 ∈ R 6 × L × L i.e. the pairwise RaptorX features described in previous subsection and/or the msa embedding covariance features ( z 3 ∈ R 256 × L × L ) described in the next subsection.”
BackgroundMethodInfluential
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Correlated mutations and residue contacts in proteins
Göbel, U., Sander, C., Schneider, R., & Valencia, A. (1994) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.340180402 · Cited by 863 · cited 6× · see citing works
“Out of the possible random perturbations to a sequence, evolution is biased toward selecting those that are consistent with fitness (Göbel et al., 1994).”
BackgroundInfluential
The Universal Protein Resource (UniProt)
The UniProt Consortium (2007) · Nucleic Acids Research · doi:10.1093/nar/gkm895 · Cited by 824 · see citing works
Mutation effects predicted from sequence co-variation
Hopf, T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. P. I., Springer, M., Sander, C., & Marks, D. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3769 · Cited by 775 · cited 2× · see citing works
“…properties that affect ﬁtness directly, such as activity, stability, structure, binding, and other properties under selection (Hormoz, 2013; Hopf et al., 2017) as well as by contributions from phylogenetic bias (Gabaldon, 2007), experimental and selection biases (Wang et al., 2019;…”
Background
SCOPe: Structural Classification of Proteins--extended, integrating SCOP and ASTRAL data and classification of new structures
Fox, N. K., Brenner, S. E., & Chandonia, J.-M. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1240 · Cited by 762 · cited 1× · see citing works
“We construct a dataset to evaluate remote homology detection using SCOPe (Fox et al., 2014).”
Method
PSICOV: precise structural contact prediction using sparse inverse covariance estimation on large multiple sequence alignments
Jones, D. T., Buchan, D. W. A., Cozzetto, D., & Pontil, M. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr638 · Cited by 760 · see citing works
Untitled work
(2017) · International Conference on Machine Learning · Cited by 758 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 1× · see citing works
“We use a deep bidirectional Transformer encoder model (Devlin et al., 2018; Vaswani et al., 2017) and process data at the character-level, corresponding to individual amino-acids in our application.”
Method
Uniclust databases of clustered and deeply annotated protein sequences and alignments
Mirdita, M., von den Driesch, L., Galiez, C., Martin, M. J., Söding, J., & Steinegger, M. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw1081 · Cited by 710 · cited 1× · see citing works
“MSAs for the baselines are generated from the UniClust30 (Mirdita et al., 2017) database using 3 iterations of search by Hhblits.”
Method
Iterated profile searches with PSI-BLAST—a tool for discovery in protein databases
Altschul, S. F., & Koonin, E. V. (1998) · Trends in Biochemical Sciences · doi:10.1016/s0968-0004(98)01298-5 · Cited by 679 · cited 1× · see citing works
“To do well at this prediction task, the learned representations must encode the underlying factors that inﬂuence sequence variation in the data.”
Background
Critical assessment of methods of protein structure prediction (CASP) — round x
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2013) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24452 · Cited by 673 · see citing works
Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., & Aurell, E. (2013) · Physical Review E · doi:10.1103/physreve.87.012707 · Cited by 661 · see citing works
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
Cuff, J. A., & Barton, G. J. (1999) · Proteins Structure Function and Bioinformatics · doi:10.1002/(sici)1097-0134(19990301)34:4 · Cited by 660 · see citing works
Conformational preferences of amino acids in globular proteins
Levitt, M. (1978) · Biochemistry · doi:10.1021/bi00613a026 · Cited by 654 · cited 1× · see citing works
“Local biochemical and structural contexts are reflected in preferences for distinct classes of amino acids (Levitt, 1978).”
Background
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · cited 1× · see citing works
“If representation space is organized linearly by species, then adding the difference in average representations to a protein in species s will recover the corresponding protein in species t .”
Background
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601 · cited 1× · see citing works
“The generative picture can be extended to include latent variables parameterized by neural networks that capture higher order interactions than pairwise (Riesselman et al., 2018).”
Background
Untitled work
(2019) · doi:10.18653/v1/n19-1419 · Cited by 601 · cited 3× · see citing works
“…To learn the distance projection used in the phylogenetic organization analysis, we applied an adapted version of the technique of Hewitt & Manning (2019), originally employed toward learning a natural language parse-tree encoding; we modify the technique to instead learn a phylogenetic…”
MethodInfluential
Better prediction of functional effects for sequence variants
Hecht, M., Bromberg, Y., & Rost, B. (2015) · BMC Genomics · doi:10.1186/1471-2164-16-s8-s1 · Cited by 598 · cited 1× · see citing works
“Computational variant effect predictors are useful for assessing the effect of point mutations (Gray et al., 2018; Adzhubei et al., 2013; Kumar et al., 2009; Hecht et al., 2015; Rentzsch et al., 2018).”
Unsupervised Machine Translation Using Monolingual Corpora Only
Lample, G., Conneau, A., Denoyer, L., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1711.00043 · Cited by 584 · see citing works
NetSurfP-2.0: improved prediction of protein structural features by integrated deep learning
Klausen, M., Jespersen, M. C., Nielsen, H., Jensen, K. K., Jurtz, V., Sønderby, C., Sommer, M., Winther, O., Nielsen, M., Petersen, B., et al. (2018) · bioRxiv · doi:10.1002/prot.25674 · Cited by 564 · see citing works
Protein Secondary Structure Prediction Using Deep Convolutional Neural Fields
Wang, S., Peng, J., Ma, J., & Xu, J. (2016) · Scientific Reports · doi:10.1038/srep18962 · Cited by 556 · cited 2× · see citing works
“For calibration, current state-of-the-art on the same dataset is 84 % on the Q3 (3-class) task (Wang et al., 2016) or 70.5% on Q8 (8-class) task (Drori et al., 2018) using a technique considerably more sophisticated than linear projection.”
MethodInfluentialSelf-citation
Protein structure determination using metagenome sequence data
Ovchinnikov, S., Park, H., Varghese, N., Huang, P.-S., Pavlopoulos, G. A., Kim, D. E., Kamisetty, H., Kyrpides, N. C., & Baker, D. (2017) · Science · doi:10.1126/science.aah4043 · Cited by 520 · cited 1× · see citing works
“Contacts predicted from evolutionary data have enabled large-scale computational prediction of protein structures (Ovchinnikov et al., 2017).”
Background
Critical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506 · cited 1× · see citing works
“We evaluate models on the CB513 test set (Cuff & Barton, 1999) and the CASP13 domains (Kryshtafovych et al., 2019).”
Method
CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations
Seemayer, S., Gruber, M., & Söding, J. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu500 · Cited by 416 · cited 2× · see citing works
“(ii) unsupervised contacts predicted by the CCMpred implementation (Seemayer et al., 2014) of direct coupling analysis.”
Background
Distance-based protein folding powered by deep learning
Xu, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1821309116 · Cited by 401 · cited 1× · see citing works
“For inputs, these methods use raw frequency and covariance statistics, or parameters of maximum-entropy sequence models ﬁtted on MSAs (Jones & Kandathil, 2018; Xu, 2018; Senior et al., 2018).”
Method
Expanding functional protein sequence spaces using generative adversarial networks
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00310-5 · Cited by 388 · see citing works
The crystal structure of MarR, a regulator of multiple antibiotic resistance, at 2.3 A resolution.
Alekshun, M. N., Levy, S. B., Mealy, T. R., Seaton, B. A., & Head, J. W. (2001) · Nature Structural Biology · doi:10.1038/90429 · Cited by 386 · cited 2× · see citing works
“Figure 6: Pre-trained Transformer models accurately predict contacts for chain A of the MarR protein (PDB: 1JGS; Alekshun et al., 2001).”
BackgroundMethod
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338 · see citing works
Critical assessment of methods of protein structure prediction (CASP)—Round XII
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25415 · Cited by 327 · cited 1× · see citing works
“The Test (59) and CASP11 (60) test sets evaluate with sequence identity holdout at 25%, the CASP12 (61) test set implements a temporal holdout with the structural training data, and the CASP13 (58) experiment implements a full temporal holdout of both the pretraining and training data.”
Learning generative models for protein fold families
Balakrishnan, S., Kamisetty, H., Carbonell, J. G., Lee, S., & Langmead, C. J. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.22934 · Cited by 316 · see citing works
Learned protein embeddings for machine learning
Yang, K., Wu, Z., Bedbrook, C. N., & Arnold, F. H. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty178 · Cited by 316 · cited 1× · see citing works
“In contrast to models that are based on recurrent or convolutional neural networks, the Transformer makes no assumptions on the ordering of the input and instead uses position embeddings.”
Sharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · see citing works
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · arXiv · doi:10.48550/arxiv.1805.04623 · Cited by 302 · see citing works
Correlation of co-ordinated amino acid substitutions with function in viruses related to tobacco mosaic virus
Altschuh, D., Lesk, A. M., Bloomer, A. C., & Klug, A. (1987) · Journal of Molecular Biology · doi:10.1016/0022-2836(87)90352-4 · Cited by 284 · cited 1× · see citing works
“The idea that biological function and structure are recorded in the statistics of protein sequences selected through evolution has a long history (Yanofsky et al., 1964; Altschuh et al., 1987; 1988).”
Background
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 282 · see citing works
Coevolutionary Landscape Inference and the Context-Dependence of Mutations in Beta-Lactamase TEM-1
Figliuzzi, M., Jacquier, H., Schug, A. H., Tenaillon, O., & Weigt, M. (2015) · Molecular Biology and Evolution · doi:10.1093/molbev/msv211 · Cited by 263 · cited 1× · see citing works
“…inductive bias of the models and frequency biases in the data, we compare against (1) untrained contextual language models: an untrained LSTM, and the Transformer at its initial state prior to training; and (2) a frequency baseline that maps a sequence to a vector of normalized amino acid counts.”
Method
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
(2018) · doi:10.18653/v1/n18-2 · Cited by 263 · see citing works
Critical assessment of methods of protein structure prediction: Progress and new directions in round XI
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2016) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25064 · Cited by 226 · cited 1× · see citing works
“The Test (59) and CASP11 (60) test sets evaluate with sequence identity holdout at 25%, the CASP12 (61) test set implements a temporal holdout with the structural training data, and the CASP13 (58) experiment implements a full temporal holdout of both the pretraining and training data.”
Quantification of the effect of mutations using a global probability model of natural sequence variationPreprint
Hopf, T. A., Ingraham, J., Poelwijk, F. J., Springer, M., Sander, C., & Marks, D. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.04612 · Cited by 224 · cited 2× · see citing works
“…properties that affect ﬁtness directly, such as activity, stability, structure, binding, and other properties under selection (Hormoz, 2013; Hopf et al., 2017) as well as by contributions from phylogenetic bias (Gabaldon, 2007), experimental and selection biases (Wang et al., 2019;…”
Background
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219 · cited 1× · see citing works
“In contrast to models that are based on recurrent or convolutional neural networks, the Transformer makes no assumptions on the ordering of the input and instead uses position embeddings.”
Cloze-driven Pretraining of Self-attention Networks
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., & Auli, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1539 · Cited by 215 · cited 1× · see citing works
“The ability to learn such representations improves signiﬁcantly with larger training datasets (Baevski et al., 2019; Radford et al., 2019).”
Background
Generating functional protein variants with variational autoencoders
Hawkins‐Hooker, A., Depardieu, F., Baur, S., Couairon, G., Chen, A., & Bikard, D. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008736 · Cited by 211 · see citing works
Quantitative Missense Variant Effect Prediction Using Large-Scale Mutagenesis Data
Gray, V. E., Hause, R. J., Luebeck, J., Shendure, J., & Fowler, D. M. (2017) · Cell Systems · doi:10.1016/j.cels.2017.11.003 · Cited by 205 · see citing works
Selection Forces and Constraints on Retroviral Sequence Variation
Overbaugh, J. M., & Bangham, C. R. M. (2001) · Science · doi:10.1126/science.1059128 · Cited by 200 · cited 1× · see citing works
“…and other properties under selection (Hormoz, 2013; Hopf et al., 2017) as well as by contributions from phylogenetic bias (Gabaldon, 2007), experimental and selection biases (Wang et al., 2019; Overbaugh & Bangham, 2001), and sources of noise such as random genetic drift (Kondrashov et al., 2003).”
Background
UDSMProt: universal deep sequence models for protein classification
Strodthoff, N., Wagner, P., Wenzel, M., & Samek, W. (2019) · bioRxiv · doi:10.1093/bioinformatics/btaa003 · Cited by 200 · see citing works
Origins of coevolution between residues distant in protein 3D structures
Anishchenko, I., Ovchinnikov, S., Kamisetty, H., & Baker, D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1702664114 · Cited by 185 · cited 2× · see citing works
“Contacts can be inferred through evolutionary correlations between residues in a sequence family (Marks et al., 2011; Anishchenko et al., 2017).”
Background
High precision in protein contact prediction using fully convolutional neural networks and minimal sequence features
Jones, D. T., & Kandathil, S. M. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty341 · Cited by 184 · cited 1× · see citing works
“We use a dilated convolutional residual network similar to recent state-of-the-art methods for tertiary structure prediction (Xu, 2018; Jones & Kandathil, 2018; Senior et al., 2018).”
Method
DeepSF: deep convolutional neural network for mapping protein sequences to folds
Hou, J., Adhikari, B., & Cheng, J. J. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx780 · Cited by 179 · see citing works
Sequence comparison and protein structure prediction
Dunbrack, R. L. (2006) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2006.05.006 · Cited by 175 · cited 1× · see citing works
“Figure 2b visualizes the projected representations of four orthologous groups across four species.”
Background
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language ModelPreprint
Wang, A., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04094 · Cited by 174 · cited 1× · see citing works
“The models we have trained can be used to generate new sequences (Wang & Cho, 2019”
Method
VideoBERT: A Joint Model for Video and Language Representation LearningPreprint
Sun, C., Myers, A., Vondrick, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01766 · Cited by 148 · see citing works
Phage-assisted continuous evolution of proteases with altered substrate specificity
Packer, M. S., Rees, H. A., & Liu, D. R. (2017) · Nature Communications · doi:10.1038/s41467-017-01055-9 · Cited by 133 · cited 1× · see citing works
“…function (Romero & Arnold, 2009), inform protein design using a combination of activating mutants (Hu et al., 2018), and make rational substitutions to optimize protein properties such as substrate speciﬁcity (Packer et al., 2017), stability (Tan et al., 2014), and binding (Ricatti et al., 2019).”
Background
Correlated mutations in models of protein sequences: phylogenetic and structural effects
Lapedes, A. S., Giraud, B., Liu, L., & Stormo, G. D. (1999) · Institute of Mathematical Statistics Lecture Notes - Monograph Series · doi:10.1214/lnms/1215455556 · Cited by 124 · see citing works
Deep Supervised and Convolutional Generative Stochastic Network for Protein Secondary Structure PredictionPreprint
Zhou, J., & Troyanskaya, O. G. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.1347 · Cited by 123 · see citing works
The effect of long‐range interactions on the secondary structure formation of proteins
Kihara, D. (2005) · Protein Science · doi:10.1110/ps.051479505 · Cited by 98 · cited 1× · see citing works
“It is well known that structural properties of protein sequences are reﬂected in long-range dependencies (Kihara, 2005).”
Background
Protein Structure Relationships Revealed by Mutational Analysis
Yanofsky, C., Horn, V., & Thorpe, D. (1964) · Science · doi:10.1126/science.146.3651.1593 · Cited by 98 · cited 1× · see citing works
“The idea that biological function and structure are recorded in the statistics of protein sequences selected through evolution has a long history (Yanofsky et al., 1964; Altschuh et al., 1987; 1988).”
Background
Coordinated amino acid changes in homologous protein families
Altschuh, D., Vernet, T., Berti, P. J., Moras, D., & Nagai, K. (1988) · Protein Engineering Design and Selection · doi:10.1093/protein/2.3.193 · Cited by 96 · cited 1× · see citing works
“The idea that biological function and structure are recorded in the statistics of protein sequences selected through evolution has a long history (Yanofsky et al., 1964; Altschuh et al., 1987; 1988).”
Background
MRFalign: Protein Homology Detection through Alignment of Markov Random Fields
Ma, J., Wang, S., Wang, Z., & Xu, J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003500 · Cited by 90 · see citing works
Self-citation
On the Mechanism of the Reaction Catalyzed by Glucose 6-Phosphate Dehydrogenase,
Cosgrove, M. S., Naylor, C. E., Paludan, S. R., et al. (1998) · Biochemistry · doi:10.1021/bi972069y · Cited by 88 · cited 1× · see citing works
“(b) illustrates the 3-class representation projections on representative proteins (PDB IDs 2DPG and 1BDJ; Cosgrove et al., 1998; Kato et al., 1999) selected for dissimilarity with the training dataset, having sequence identity no higher than 0.”
Background
Protein sequence comparison and fold recognition: progress and good-practice benchmarking
Söding, J., & Remmert, M. (2011) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2011.03.005 · Cited by 80 · cited 1× · see citing works
“Following standard practices (45), we exclude Rossmann-like folds (c.”
Method
Graphical Models of Residue Coupling in Protein Families
Thomas, J., Ramakrishnan, N., & Bailey-Kellogg, C. (2008) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2007.70225 · Cited by 77 · cited 1× · see citing works
“Direct coupling analysis (Lapedes et al., 1999; Thomas et al., 2008; Weigt et al., 2009) infers constraints on the structure of a protein by fitting a generative model in the form of a Markov Random Field (MRF) to the sequences in the protein’s MSA.”
Method
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA.
Belle, V., den Broeck, G. V., & Passerini, A. (2016) · National Conference on Artificial Intelligence · Cited by 75 · see citing works
Amino acid composition of proteins reduces deleterious impact of mutations
Hormoz, S. (2013) · Scientific Reports · doi:10.1038/srep02919 · Cited by 68 · see citing works
Designing Feature-Controlled Humanoid Antibody Discovery Libraries Using Generative Adversarial NetworksPreprint
Amimeur, T., Shaver, J. M., Ketchem, R. R., Taylor, J. A., Clark, R. H., Smith, J., Citters, D. V., Siska, C. C., Smidt, P., Sprague, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.12.024844 · Cited by 64 · cited 1× · see citing works
“Classical methods that aim to detect pairwise dependencies in multiple sequence alignments are able to model entire sequences.”
Background
DEEPCON: protein contact prediction using dilated convolutional neural networks with dropout
Adhikari, B. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz593 · Cited by 49 · see citing works
Evolution of Proteins and Proteomes: A Phylogenetics Approach
Gabaldón, T. (2005) · Evolutionary Bioinformatics · doi:10.1177/117693430500100004 · Cited by 45 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42 · see citing works
Impact of selection, mutation rate and genetic drift on human genetic variation
Sunyaev, S. R. (2003) · Human Molecular Genetics · doi:10.1093/hmg/ddg359 · Cited by 39 · cited 1× · see citing works
“, 2019; Overbaugh & Bangham, 2001), and sources of noise such as random genetic drift (Kondrashov et al., 2003).”
Background
Distance-based Protein Folding Powered by Deep LearningPreprint
Xu, J. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/465955 · Cited by 39 · see citing works
Protein Design and Variant Prediction Using Autoregressive Generative ModelsPreprint
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E. P., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/757252 · Cited by 39 · see citing works
Sequence-based protein stabilization in the absence of glycosylation
Tan, N. Y. J., Bailey, U., Jamaluddin, M. F. B., Mahmud, S. H. B., Raman, S. C., & Schulz, B. L. (2014) · Nature Communications · doi:10.1038/ncomms4099 · Cited by 38 · cited 2× · see citing works
“, 2017), stability (Tan et al., 2014), and binding (Ricatti et al.”
Background
Generating functional protein variants with variational autoencodersPreprint
Hawkins‐Hooker, A., Depardieu, F., Baur, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.07.029264 · Cited by 34 · see citing works
Structure and dynamics of the B12‐binding subunit of glutamate mutase from Clostridium cochlearium
Hoffmann, B., Konrat, R., Bothe, H., et al. (1999) · European Journal of Biochemistry · doi:10.1046/j.1432-1327.1999.00482.x · Cited by 32 · see citing works
Expanding functional protein sequence space using generative adversarial networksPreprint
Repecka, D., Jauniškis, V., Karpus, L., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/789719 · Cited by 31 · cited 1× · see citing works
“It is well known that structural properties of protein sequences are reﬂected in long-range dependencies (Kihara, 2005).”
Background
Correlated mutations in protein sequences: Phylogenetic and structural effects
Los Alamos National Lab., T. D., Lapedes, A. S., Liu, L. C., GIRAUD, B. G., Liu, L., & Stormo, G. D. (1998) · doi:10.2172/296863 · Cited by 25 · see citing works
High Quality Prediction of Protein Q8 Secondary Structure by Diverse Neural Network ArchitecturesPreprint
Drori, I., Dwivedi, I., Shrestha, P., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07143 · Cited by 25 · cited 1× · see citing works
“For calibration, current state-of-the-art on the same dataset is 84 % on the Q3 (3-class) task (Wang et al., 2016) or 70.5% on Q8 (8-class) task (Drori et al., 2018) using a technique considerably more sophisticated than linear projection.”
MethodInfluential
Revealing evolutionary constraints on proteins through sequence analysis
Wang, S., Bitbol, A., & Wingreen, N. S. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007010 · Cited by 25 · cited 2× · see citing works
“, 2017) as well as by contributions from phylogenetic bias (Gabaldon, 2007), experimental and selection biases (Wang et al., 2019; Overbaugh & Bangham, 2001), and sources of noise such as random genetic drift (Kondrashov et al.”
Background
Effects of point mutations in the binding pocket of the mouse major urinary protein MUP20 on ligand affinity and specificity
Ricatti, M. J., Acquasaliente, L., Ribaudo, G., Filippis, V. D., Bellini, M., Llovera, R. E., Barollo, S., Pezzani, R., Zagotto, G., Persaud, K. C., et al. (2019) · Scientific Reports · doi:10.1038/s41598-018-36391-3 · Cited by 23 · cited 1× · see citing works
“…function (Romero & Arnold, 2009), inform protein design using a combination of activating mutants (Hu et al., 2018), and make rational substitutions to optimize protein properties such as substrate speciﬁcity (Packer et al., 2017), stability (Tan et al., 2014), and binding (Ricatti et al., 2019).”
Background
End-to-end multitask learning, from protein language to protein features without alignmentsPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., & Rost, B. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/864405 · Cited by 17 · cited 1× · see citing works
“One key factor affecting the performance of LSTMs on these tasks is the path lengths that must be traversed by forward activation and backward gradient signals in the network (Hochreiter et al., 2001).”
Background
Structure of the histidine-containing phosphotransfer (HPt) domain of the anaerobic sensor protein ArcB complexed with the chemotaxis response regulator CheY
Kato, M., Shimizu, T., Mizuno, T., et al. (1999) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s0907444999005053 · Cited by 16 · see citing works
Evolutionary Context-Integrated Deep Sequence Modeling for Protein Engineering
Luo, Y., Vo, L. T., Ding, H., Su, Y., Liu, Y., Qian, W. W., Zhao, H., & Peng, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45257-5_30 · Cited by 14 · see citing works
Evolutionary context-integrated deep sequence modeling for protein engineeringPreprint
Luo, Y., Vo, L. T., Ding, H., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.01.16.908509 · Cited by 13 · cited 1× · see citing works
“Subsequent to our preprint, related works have built on its exploration of protein sequence modeling, exploring generative models (71, 72), internal representations of Transformers (73), and applications of representation learning and generative modeling such as classification (74, 75), mutational effect prediction (8…”
Background
MRFalign: Protein Homology Detection through Alignment of Markov Random Fields
Ma, J., Wang, S., Wang, Z., & Xu, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-05269-4_13 · Cited by 6 · see citing works
Self-citation
DEEPCON: Protein Contact Prediction using Dilated Convolutional Neural Networks with DropoutPreprint
Adhikari, B. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/590455 · Cited by 5 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · cited 5×
“…a form of unsupervised learning in which context within the text is used to predict missing words, have been shown to materialize representations of word meaning that can generalize across problems (Collobert & Weston, 2008; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2018).”
BackgroundMethodInfluential

Flagged first, then by citations

Research works printed but not linked: 23

Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., and Aurell, E. Improved contact prediction in proteins: using pseu- dolikelihoods to infer potts models. Physical Review E, 87(1):012707, 2013b.
Mikolov, T., Sutskever, I., Deoras, A., Le, H.-S., Kom- brink, S., and Cernocky, J. Subword language modeling with neural networks. preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf), 8, 2012.
Riesselman, A. J., Shin, J.-E., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., and Marks, D. S. Accelerating protein design using autore- gressive generative models. bioRxiv, pp. 757252, 2019.
Balakrishnan, S., Kamisetty, H., Carbonell, J. G., Lee, S.-I., and Langmead, C. J. Learning generative models for protein fold families. Proteins: Structure, Function, and Bioinformatics, 79(4):1061-1078, 2011.
Bateman, A., Heger, A., Sonnhammer, E. L. L., Mistry, J., Clements, J., Tate, J., Hetherington, K., Holm, L., Punta, M., Coggill, P., Eberhardt, R. Y., Eddy, S. R., and Finn, R. D. Pfam: the protein families database. Nucleic Acids Research, 42(D1):D222-D230, 11 2013. ISSN 0305-1048. doi: 10.1093/nar/gkt1223. · DOI · Paper page
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. CoRR, abs/1810.04805, 2018b. URL http://arxiv.org/abs/1810.04805. · arXiv
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., and Aurell, E. Improved contact prediction in proteins: using pseu- dolikelihoods to infer potts models. Physical Review E, 87(1):012707, 2013.
Fox, N. K., Brenner, S. E., and Chandonia, J.-M. SCOPe: Structural Classification of Proteins-extended, inte- grating SCOP and ASTRAL data and classification of new structures. Nucleic Acids Research, 42(D1): D304-D309, January 2014. ISSN 0305-1048, 1362- 4962. doi: 10.1093/nar/gkt1240. URL https:// academic.oup.com/nar/article-lookup/ doi/10.1093/nar/gkt1240. · DOI · Paper page
Gray, V. E., Hause, R. J., Luebeck, J., Shendure, J., and Fowler, D. M. Quantitative missense variant effect pre- diction using large-scale mutagenesis data. Cell systems, 6(1):116-124, 2018.
Huerta-Cepas, J., Forslund, S. K., Bork, P., Hernández- Plaza, A., von Mering, C., Szklarczyk, D., Heller, D., Cook, H., Jensen, L., Mende, D. R., Letunic, I., and Rattei, T. eggNOG 5.0: a hierarchical, functionally and phylogenetically annotated orthology resource based on 5090 organisms and 2502 viruses. Nucleic Acids Re- search, 47(D1):D309-D314, 11 2018. ISSN 0305-1048. doi: 10.1093/nar/gky1085. · DOI
Johnson, J., Douze, M., and Jégou, H. Billion-scale similar- ity search with gpus. CoRR, abs/1702.08734, 2017. URL http://arxiv.org/abs/1702.08734. · arXiv
Klausen, M. S., Jespersen, M. C., Nielsen, H., Jensen, K. K., Jurtz, V. I., Sonderby, C. K., Sommer, M. O. A., Winther, O., Nielsen, M., Petersen, B., and Marcatili, P. NetSurfP- 2.0: Improved prediction of protein structural features by integrated deep learning. Proteins, 87(6):520-527, 06 2019.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Ap- proach. 7 2019. URL http://arxiv.org/abs/ 1907.11692.
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pp. 2227- 2237, 2018. URL https://aclanthology.info/ papers/N18-1202/n18-1202.
Remmert, M., Biegert, A., Hauser, A., and Söding, J. Hh- blits: lightning-fast iterative protein sequence searching by hmm-hmm alignment. Nature Methods, 9:173 EP, 12 2011. doi: 10.1038/nmeth.1818. · DOI · Paper page
Riesselman, A. J., Ingraham, J. B., and Marks, D. S. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, 15(10):816-822, 2018. ISSN 1548-7105. doi: 10.1038/s41592-018-0138-4. · DOI · Paper page
Seemayer, S., Gruber, M., and Söding, J. Ccmpred-fast and precise prediction of protein residue-residue contacts from correlated mutations. Bioinformatics, 30(21):3128- 3130, 2014.
Söding, J. and Remmert, M. Protein sequence comparison and fold recognition: progress and good-practice bench- marking. Current opinion in structural biology, 21(3): 404-411, 2011.
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., and Consortium, U. Uniref clusters: a comprehen- sive and scalable alternative for improving sequence sim- ilarity searches. Bioinformatics, 31(6):926-932, 2015.
Thomas, J., Ramakrishnan, N., and Bailey-Kellogg, C. Graphical models of residue coupling in protein fami- lies. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 5(2):183-197, 2008.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Atten- tion is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017b.
Wang, S., Sun, S., Li, Z., Zhang, R., and Xu, J. Accu- rate de novo prediction of protein contact map by ultra- deep learning model. PLoS computational biology, 13(1): e1005324, 2017.
Weigt, M., White, R. A., Szurmant, H., Hoch, J. A., and Hwa, T. Identification of direct residue contacts in protein- protein interaction by message passing. Proceedings of the National Academy of Sciences, 106(1):67-72, 2009.

Sorted by citations

Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
“In parallel, the success of attention-based networks for language processing 52 and, more recently, computer vision 31,53 has inspired the exploration of attention-based methods for interpreting protein sequences 54–56 .”
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
“[14] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C.”
BackgroundSelf-citation
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Self-citation
SignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01156-3 · Cited by 2,999 · see citing works
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
“As previously established for another protein LM [24], the t-SNE projections (Fig.”
BackgroundMethodResultInfluential
Accurate proteome-wide missense variant effect prediction with AlphaMissense
Cheng, J., Novati, G., Pan, J., Bycroft, C., Žemgulytė, A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al. (2023) · Science · doi:10.1126/science.adg7492 · Cited by 2,163 · see citing works
“After pretraining, the masked language modeling head can already be used for variant effect prediction by computing the log-likelihood ratio between the reference and alternative amino acid probabilities, as done in MSA Transformer ( 27 ) and Evolutionary Scale Modeling [ESM ( 28 )].”
Method
SignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000524414 · Cited by 1,905 · see citing works
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823 · see citing works
DeepTMHMM predicts alpha and beta transmembrane proteins using deep neural networksPreprint
Hallgren, J., Tsirigos, K. D., Pedersen, M. D., Armenteros, J. J. A., Marcatili, P., Nielsen, H., Krogh, A., & Winther, O. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.08.487609 · Cited by 1,528 · see citing works
A survey of transformers
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422 · see citing works
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078 · see citing works
Machine-learning-guided directed evolution for protein engineering
Yang, K., Wu, Z., & Arnold, F. H. (2019) · Nature Methods · doi:10.1038/s41592-019-0496-6 · Cited by 1,082 · see citing works
“However, more sophisticated continuous vector encodings of sequences can be learned from patterns in unlabeled sequences 52,70–76 or from structural information 77 .”
Background
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · see citing works
“Existing works show that, other things being equal, larger models and additional pretraining computation time correlates with improved model performance [2, 5, 20].”
BackgroundMethodInfluential
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795 · see citing works
“Given both the growth of proteinsequence databases and the lack of annotation for a signi ﬁ cant part of the protein space, protein sequences have become great candidates for unsupervised training 4,10,11 and now offer the opportunity to encode and generate protein sequences.”
BackgroundMethod
Disease variant prediction with deep generative models of evolutionary data
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., & Marks, D. S. (2021) · Nature · doi:10.1038/s41586-021-04043-8 · Cited by 791 · see citing works
Simulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937 · see citing works
Self-citation
DeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 723 · see citing works
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635 · see citing works
Self-citation
Automated model building and protein identification in cryo-EM maps
Jamali, K., Käll, L., Zhang, R., Brown, A., Kimanius, D., & Scheres, S. H. W. (2024) · Nature · doi:10.1038/s41586-024-07215-4 · Cited by 686 · see citing works
“The sequence module performs cross-attention for each residue with the user-provided amino acid sequences, which are embedded using the pretrained protein language model ESM-1b 31 .”
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
“…2018), document generation/summarization, time series prediction, generative modeling (e.g. image generation (Parmar et al., 2018)), music generation (Huang et al., 2019), and bioinformatics (Rives et al., 2019; Madani et al., 2020; Ingraham et al., 2019; Elnaggar et al., 2019; Du et al., 2020).”
Background
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · see citing works
“Similar to the results described in [51], our projection of the embedding space confirmed that the model successfully captured bio-chemical and bio-physical properties on the most fine-grained level, i.e. the 20 standard amino acids (Fig.”
BackgroundResultInfluential
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 656 · see citing works
“These protein LMs, when applied to downstream tasks using transfer learning, demonstrated the ability to compete with and even outperform previous methods for tasks such as predicting protein structure 3,4 and function 6,7 , even in data-scarce regimens 8 .”
Large-scale chemical language representations capture molecular structure and properties
Ross, J., Belgodere, B., Chenthamarakshan, V., Padhi, I., Mroueh, Y., & Das, P. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00580-7 · Cited by 590 · see citing works
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
Learning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483 · see citing works
DeepGOPlus: improved protein function prediction from sequence
Kulmanov, M., & Hoehndorf, R. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz595 · Cited by 476 · see citing works
Efficient evolution of human antibodies from general protein language models
Hie, B. L., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P., Tang, S.-G., Wu, W., Pak, J., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 452 · see citing works
“Because we used general language models 19,20 , trained on non-redundant sequence datasets that are meant to represent variation across all natural proteins 23 , these models can only learn more general evolutionary rules than could a model trained specifically on antibody sequences 24–27 or a…”
Illuminating protein space with a programmable generative model
Ingraham, J. B., Baranov, M., Costello, Z., Barber, K. W., Wang, W., Ismail, A., Frappier, V., Lord, D. M., Ng-Thow-Hing, C., Van Vlack, E. R., et al. (2023) · Nature · doi:10.1038/s41586-023-06728-8 · Cited by 444 · see citing works
Enzyme function prediction using contrastive learning
Yu, T., Cui, H., Li, J. C., Luo, Y., Jiang, G., & Zhao, H. (2023) · Science · doi:10.1126/science.adf2465 · Cited by 451 · see citing works
Chai-1: Decoding the molecular interactions of life
Discovery, C., Boitreaud, J., Dent, J., McPartlon, M., Meier, J., Reis, V., Rogozhnikov, A., & Wu, K. (2024) · doi:10.1101/2024.10.10.615955 · Cited by 517 · see citing works
Self-citation
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455 · see citing works
“Language models were first developed as a means to extract semantic information from a sequence of words (a key requirement for natural language processing) 26 .”
BackgroundMethod
Learning inverse folding from millions of predicted structuresPreprint
Hsu, C., Verkuil, R., Liu, J. J., Lin, Z., Hie, B., Sercu, T., Lerer, A. K., & Rives, A. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 434 · see citing works
“…2018; Eguchi et al., 2020), jointly generate structures and sequences (An-ishchenko et al., 2021; Wang et al., 2021), or model sequences directly (Rives et al., 2021; Madani et al., 2021; Shin et al., 2021; Gligorijevic et al., 2021; Bryant et al., Augmenting inverse folding with predicted…”
BackgroundSelf-citation
Genome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 459 · see citing works
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
Expanding functional protein sequence spaces using generative adversarial networks
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00310-5 · Cited by 388 · see citing works
“On the other hand, machine learning methods are not limited by the number of sequence variations they can process 14–16 and, instead of depending on a semi-random local search process, infer protein properties 15,17 and function 16,18 directly from the amino-acid sequence.”
Background
How to build the virtual cell with artificial intelligence: Priorities and opportunities
Bunne, C., Roohani, Y., Rosen, Y., Gupta, A., Zhang, X., Roed, M., Alexandrov, T., AlQuraishi, M., Brennan, P., Burkhardt, D. B., et al. (2024) · Cell · doi:10.1016/j.cell.2024.11.015 · Cited by 393 · see citing works
High-resolution <i>de novo</i> structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392 · see citing works
“…Helixon US Inc, USA ‡ Westlake Laboratory of Life Sciences and Biomedicine, Hangzhou, Zhejiang 310012, China § Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139 ¶ Correspondence 1 sufficient to predict the protein 3D structures ( 23 ).”
BackgroundMethodSelf-citation
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
MSA Transformer
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350 · see citing works
Self-citation
NetSurfP-3.0: accurate and fast prediction of protein structural features by protein language models and deep learning
Høie, M. H., Kiehl, E. N., Petersen, B., Nielsen, M., Winther, O., Nielsen, H., Hallgren, J., & Marcatili, P. (2022) · Nucleic Acids Res. · doi:10.1093/nar/gkac439 · Cited by 325 · see citing works
“In doing so, they generate complex numerical representations of protein sequences (also referred to as sequence embeddings), that can be exploited to perform downstream prediction tasks, thus bypassing the need to generate computationally expensive MSAs.”
BackgroundMethodInfluential
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338 · see citing works
“Sequence-only language models have since been trained through autoregressive or autoencoding self-supervision objectives for both discriminative and generative tasks, for example, using LSTMs or Transformer-based architectures [2, 9, 60, 64].”
Method
The language of proteins: NLP, machine learning & protein sequences
Ofer, D., Brandes, N., & Linial, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.03.022 · Cited by 345 · see citing works
“As of the present day, ﬁne-tuning of pretrained deep models have shown considerable promise for improving our ability to solve problems, even on small data [79,83,19].”
BackgroundInfluential
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
Self-citation
Using deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332 · see citing works
Protein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 339 · see citing works
“Prior work has demonstrated that protein language models can learn structural features from sequence pre-training 55 alone (17, 33).”
Background
Utilizing graph machine learning within drug discovery and development
Gaudelet, T., Day, B., Jamasb, A. R., Soman, J., Regep, C., Liu, G., Hayter, J. B. R., Vickers, R., Roberts, C., Tang, J., et al. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab159 · Cited by 313 · see citing works
<scp>BepiPred</scp> ‐3.0: Improved B‐cell epitope prediction using protein language models
Clifford, J. N., Høie, M. H., Deleuran, S., Peters, B., Nielsen, M., & Marcatili, P. (2022) · Protein Science · doi:10.1002/pro.4497 · Cited by 296 · see citing works
Accurate RNA 3D structure prediction using a language model-based deep learning approach
Shen, T., Hu, Z.-H., Sun, S., Liu, D., Wong, F., Wang, J.-M., Chen, J., Wang, Y., Hong, L., Xiao, J., et al. (2022) · Nature Methods · doi:10.1038/s41592-024-02487-0 · Cited by 274 · see citing works
ScanNet: an interpretable geometric deep learning model for structure-based protein binding site prediction
Tubiana, J., Schneidman-Duhovny, D., & Wolfson, H. J. (2022) · Nature Methods · doi:10.1038/s41592-022-01490-7 · Cited by 274 · see citing works
State-of-the-Art Estimation of Protein Model Accuracy Using AlphaFold
Roney, J. P., & Ovchinnikov, S. (2022) · Physical Review Letters · doi:10.1103/physrevlett.129.238101 · Cited by 281 · see citing works
Machine learning for functional protein design
Notin, P., Rollins, N. J., Gal, Y., Sander, C., & Marks, D. S. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 278 · see citing works
“This has led to a wide diversity of models, inspired by learning paradigms initially introduced in the natural language-processing literature, such as autoregressive modeling 20,52,61,141 , masked-language modeling 15,146,175,176 or seq2seq architectures 177,178 .”
Generative models for graph-based protein design
Ingraham, J., Garg, V., Barzilay, R., & Jaakkola, T. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 258 · see citing works
Rhea, the reaction knowledgebase in 2022
Bansal, P., Morgat, A., Axelsen, K. B., Muthukrishnan, V., Coudert, E., Aimo, L., Hyka‐Nouspikel, N., Gasteiger, E., Kerhornou, A., Neto, T. B., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab1016 · Cited by 251 · see citing works
Improved protein structure prediction by deep learning irrespective of co-evolution information
(2021) · doi:10.1038/s42256-021-00348-5 · Cited by 256 · see citing works
“Recently, language models such as Transformer have been used to model protein sequences[33].”
Method
Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering
Yang, J., Li, F.-Z., & Arnold, F. H. (2024) · ACS Central Science · doi:10.1021/acscentsci.3c01275 · Cited by 257 · see citing works
ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance ComputingPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Fehér, T., Angerer, C. M., Steinegger, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06225 · Cited by 305 · see citing works
“As previously established for another protein LM [24], the t-SNE projections (Fig.”
BackgroundMethodResultInfluential
Machine learning in protein structure prediction
AlQuraishi, M. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.005 · Cited by 252 · see citing works
Autonomous Discovery in the Chemical Sciences Part II: Outlook
Coley, C. W., Eyke, N. S., & Jensen, K. F. (2019) · Angewandte Chemie International Edition · doi:10.1002/anie.201909989 · Cited by 248 · see citing works
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 251 · see citing works
“Its training lasted 3 months and utilized 64 NVIDIA 80G A100 GPUs, with a computational cost similar to ESM-1b (Rives et al., 2019).”
BackgroundMethodInfluential
Bilingual language model for protein sequence and structure
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., & Rost, B. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae150 · Cited by 311 · see citing works
“…: embedding-based annotation transfer using the shortest Euclidean distance measured in embedding space of unsupervised pLMs ESM-1b [5], ProtT5 and ProstT5 with different inputs (AA=amino acid sequence input, p3Di=3Di predicted by ProstT5, 3Di=3Di from experimental structures,…”
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 239 · see citing works
Direct prediction of intrinsically disordered protein conformational properties from sequence
Lotthammer, J. M., Ginell, G. M., Griffith, D., Emenecker, R. J., & Holehouse, A. S. (2024) · Nature Methods · doi:10.1038/s41592-023-02159-5 · Cited by 228 · see citing works
DeepLoc 2.1: multi-label membrane protein type prediction using protein language models
Ødum, M. T., Teufel, F., Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae237 · Cited by 230 · see citing works
“We use sequence-based embeddings from the ProtT5 ( 6 ) and ESM-1B protein language models ( 7 ) and implement the same model architecture used by DeepLoc 2.0, performing interpretable attention pooling over sequence embeddings with discrete cosine transform regularization.”
Influential
Deep Learning in Protein Structural Modeling and Design
Gao, W., Mahajan, S. P., Sulam, J., & Gray, J. J. (2020) · Patterns · doi:10.1016/j.patter.2020.100142 · Cited by 227 · see citing works
“Rives et al.59 trained a transformer model with 670 million parameters on 86 billion amino acids across 250 million protein sequences span- ning evolutionary diversity.”
Method
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 282 · see citing works
“Recent research (Alley et al., 2019; Rives et al., 2019; Rao et al., 2019) has begun to capitalize on the much larger set of raw protein sequences by adapting state-of-the-art representation learning techniques (Devlin et al., 2018) from natural language processing (NLP) to classiﬁcation of protein…”
Background
Antibody structure prediction using interpretable deep learning
Ruffolo, J. A., Sulam, J., & Gray, J. J. (2022) · Patterns · doi:10.1016/j.patter.2021.100406 · Cited by 216 · see citing works
Fine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229 · see citing works
“Overall, pLMs have also shown to be not particularly sensitive to hyperparameter changes, neither in our work not that of others (S Sledzieski et al & JL Ferres bioRxiv doi 10.1101/2023.11.09.566187).”
Influential
UniKP: a unified framework for the prediction of enzyme kinetic parameters
Yu, H., Deng, H., He, J., Keasling, J. D., & Luo, X. (2023) · Nature Communications · doi:10.1038/s41467-023-44113-1 · Cited by 226 · see citing works
Fast and Flexible Protein Design Using Deep Graph Neural Networks
Strokach, A. V., Becerra, D., Corbi‐Verge, C., Perez‐Riba, A., & Kim, P. M. (2020) · Cell Systems · doi:10.1016/j.cels.2020.08.016 · Cited by 218 · see citing works
Genome modeling and design across all domains of life with Evo 2Preprint
Brixi, G., Durrant, M. G., Ku, J., Poli, M., Brockman, G., Chang, D., Gonzalez, G. A., King, S. B., Li, D. B., Merchant, A. T., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.18.638918 · Cited by 218 · see citing works
Self-citation
Rapid in silico directed evolution by a protein language model with EVOLVEpro.
Jiang, K.-Y., Yan, Z.-Q., Bernardo, M. D., Sgrizzi, S. R., Villiger, L., Kayabolen, A., Kim, B. J., Carscadden, J. K., Hiraizumi, M., Nishimasu, H., et al. (2024) · Science · doi:10.1126/science.adr6006 · Cited by 248 · see citing works
D-SCRIPT translates genome to phenome with sequence-based, structure-aware, genome-scale predictions of protein-protein interactions
Sledzieski, S., Singh, R., Cowen, L., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.08.010 · Cited by 208 · see citing works
Informed training set design enables efficient machine learning-assisted directed protein evolution
Wittmann, B. J., Yue, Y., & Arnold, F. H. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.008 · Cited by 208 · see citing works
Generating functional protein variants with variational autoencoders
Hawkins‐Hooker, A., Depardieu, F., Baur, S., Couairon, G., Chen, A., & Bikard, D. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008736 · Cited by 211 · see citing works
“76 Deep sequence models such as LSTMs and transformers trained on datasets spanning the entire 77 range of known sequences have been shown to learn representations which distill structural and 78 functional information from the sequence [30, 31].”
Background
Machine Learning-Guided Protein Engineering
Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlář, J., Damborský, J., Pluskal, T., Šivic, J., & Mazurenko, S. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c02743 · Cited by 214 · see citing works
Learning protein fitness models from evolutionary and assay-labeled data
Hsu, C., Nisonoff, H. M., Fannjiang, C., & Listgarten, J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01146-5 · Cited by 222 · see citing works
“We assessed a total of 13 published machine learning methods for protein fitness (1) weak-positive only learning methods that rely on probability density modeling of evolutionarily related sequences (profile HMM 27 , EVmutation 23 and DeepSequence 24 ); (2) semi-supervised learning methods that rely on unlabeled UniRe…”
BackgroundMethodInfluential
Can we predict T cell specificity with digital biology and machine learning?
Hudson, D., Fernandes, R. A., Basham, M., Ogg, G. S., & Koohy, H. (2023) · Nature reviews. Immunology · doi:10.1038/s41577-023-00835-3 · Cited by 197 · see citing works
“However, the advent of automated protein structure prediction with software programs such as RoseTTaFold, ESMFold and AlphaFold-Multimer provide potential opportunities for large-scale sequence and structure interpretations of TCR epitope specificity 63–65 .”
Method
Untitled work
(2023) · doi:10.1101/2023.12.07.570727 · Cited by 207 · see citing works
“They include UniRep [Alley et al., 2019a], the RITA suite [Hesslow et al., 2022], the ESM1 and ESM2 suite [Rives et al., 2021, Meier et al., 2021, Lin et al., 2023], VESPA [Marquet et al., 2022], the CARP suite [Yang et al., 2023a] and the ProGen2 suite [Nijkamp et al., 2022].”
BackgroundMethod
Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions
Chen, J., Hu, Z., Sun, S., Tan, Q., Wang, Y., Yu, Q., Zong, L., Hong, L., Xiao, J., Shen, T., et al. (2022) · doi:10.1101/2022.08.06.503062 · Cited by 247 · see citing works
“Inspired by recent great success in natural language processing, computer vision, and bioinformatics [64][65][66], we explore the possibilities of a general transformer architecture in RNA-related studies.”
Background
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181 · see citing works
“[7] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C.”
BackgroundSelf-citation
Learning functional properties of proteins with language models
Ünsal, S., Ataş, H., Albayrak, M., Turhan, K., Acar, A. C., & Doğan, T. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00457-9 · Cited by 196 · see citing works
“Finally, ProtALBERT, ProtT5-XL, SeqVec and ESM-1b are the best performers for the others class (Supplementary Fig.”
Influential
Modeling conformational states of proteins with AlphaFold
Sala, D., Engelberger, F., Mchaourab, H. S., & Meiler, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102645 · Cited by 182 · see citing works
DeepGOPlus: improved protein function prediction from sequence
Kulmanov, M., & Hoehndorf, R. (2021) · Computer applications in the biosciences · doi:10.1093/bioinformatics/btaa763 · Cited by 198 · see citing works
AbLang: an antibody language model for completing antibody sequences
Olsen, T. H., Moal, I. H., & Deane, C. M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac046 · Cited by 198 · see citing works
Controllable protein design with language models
Ferruz, N., & Höcker, B. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00499-z · Cited by 196 · see citing works
“In an effort led by Facebook AI, ESM-1b, an encoder Transformer trained on 250 million protein sequences, was published 48 .”
BackgroundMethod
Contrastive learning in protein language space predicts interactions between drugs and protein targets
Singh, R., Sledzieski, S., Bryson, B. D., Cowen, L., & Berger, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2220778120 · Cited by 207 · see citing works
“Specifically, we investigate the pretrained models Prose (30), ESM (76), and ProtBert (27), with default dimensions dt = 6165, 1280, and1024, respectively (SI Appendix, S2).”
ProteInfer, deep neural networks for protein functional inference
Sanderson, T., Bileschi, M. L., Belanger, D., & Colwell, L. J. (2023) · eLife · doi:10.7554/elife.80942 · Cited by 192 · see citing works
“, 2021), predicting the functional effects of mutations (Riesselman et al., 2018; Rives et al., 2021; Rao et al., 2019; Frazer et al., 2021), and protein design (Yang et al.”
Background
A general model to predict small molecule substrates of enzymes based on machine and deep learning
Kroll, A., Ranjan, S., Engqvist, M. K. M., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38347-2 · Cited by 193 · see citing works
“In this work, wego beyond the current state-of-the-artby creating maximally informative protein representations, using a customized, task-speci ﬁ c version of the ESM-1b transformer model 34 .”
BackgroundMethodInfluential
Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning
Kroll, A., Rousset, Y., Hu, X.-P., Liebrand, N. A., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-39840-4 · Cited by 186 · see citing works
UDSMProt: universal deep sequence models for protein classification
Strodthoff, N., Wagner, P., Wenzel, M., & Samek, W. (2019) · bioRxiv · doi:10.1093/bioinformatics/btaa003 · Cited by 200 · see citing works
BioPhi: A platform for antibody design, humanization, and humanness evaluation based on natural antibody repertoires and deep learning
Prihoda, D., Maamary, J., Waight, A., Juan, V., Fayadat-Dilman, L., Svozil, D., & Bitton, D. A. (2022) · mAbs · doi:10.1080/19420862.2021.2020203 · Cited by 190 · see citing works
“Multiple methods were developed with the goal of encoding meaningful compact representations of protein sequences by pre-training on raw corpuses of amino acid sequences [23][24] or MSAs [25].”
Method
Transformer-based deep learning for predicting protein properties in the life sciences
Chandra, A. A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 192 · see citing works
“Mutation prediction was also among one of the tasks in Rives et al., 2021 and Rao et al., 2019 to verify the potential of their new pre-trained Transformer models.”
BackgroundMethodResultInfluential
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 201 · see citing works
“Notably, PLM sizes have jumped from ∼ 10(6) [4] to ∼ 10(9) [10] parameters recently.”
Background
Applications of transformer-based language models in bioinformatics: a survey
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176 · see citing works
Embeddings from protein language models predict conservation and variant effects
(2021) · doi:10.1007/s00439-021-02411-y · Cited by 174 · see citing works
Computational and artificial intelligence-based methods for antibody development
Kim, J., McFee, M., Fang, Q., Abdin, O., & Kim, P. M. (2023) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2022.12.005 · Cited by 178 · see citing works
Embeddings from deep learning transfer GO annotations beyond homology
Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T., & Rost, B. (2021) · Scientific Reports · doi:10.1038/s41598-020-80786-0 · Cited by 169 · see citing works
Transformers in single-cell omics: a review and new perspectives
Szałata, A., Hrovatin, K., Becker, S., Tejada-Lapuerta, A., Cui, H., Wang, B., & Theis, F. J. (2024) · Nature Methods · doi:10.1038/s41592-024-02353-z · Cited by 176 · see citing works
“Transformers have been very impactful in the processing of sequential data, for which they were originally designed 5 , such as natural language 18 , DNA 10 or protein sequences 19 .”

What does a language model know about proteins?
Ruffolo, J. A. (2025) · Nature Methods · doi:10.1038/s41592-025-02837-6 · Cited by 1 · see citing works
Speaking to nature: a deep learning representational model of proteins ushers in protein linguistics
Bojar, D. (2019) · Synthetic Biology · doi:10.1093/synbio/ysz013 · Cited by 1 · see citing works
Modeling Protein Using Large-scale Pretrain Language ModelPreprint
Xiao, Y., Qiu, J., Li, Z., Hsieh, C., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07435 · Cited by 28 · see citing works
Language Modeling Materializes a World Model of Protein BiologyPreprint
Candido, S., Hayes, T., Derry, A., Rao, R., Lin, Z., Verkuil, R., Wu, B. Z., Lee, J. S., Bruguera, E. S., Keval, J. A., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.03.729735 · Cited by 65 · see citing works
Self-citation
Learning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483 · see citing works

Authors: Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., & Fergus, R.
Affiliations: Meta (United States) (US)
New York University (US)
Harvard University (US)
Yale University (US)
Corresponding author: Rives, Alexander
ORCID: Rives, Alexander
Meier, Joshua
Sercu, Tom
Lin, Zeming
Liu, Jason
Zitnick, C. Lawrence
Ma, Jerry
Fergus, Rob
Venue: Proceedings of the National Academy of Sciences
Publisher: openRxiv
Published: 2019-04-29
Volume: 118
Issue: 15
ISSN: 0027-8424 (ISSN-L)
DOI: 10.1101/622803
PMC: PMC8053943
OpenAlex: W3146944767
Cited by: 3,280
References: 100
Citations by year: 2026: 780 · 2025: 816 · 2024: 697 · 2023: 448 · 2022: 305 · 2021: 161 · 2020: 51 · 2019: 18 · 2012: 2
FWCI: 126.76
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by-nc-nd
Locations: Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
PubMed Central (publishedVersion, cc-by-nc-nd)
PubMed (publishedVersion)
Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
University of Chicago
Funders: National Science Foundation (1339362)
Topics: Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
Keywords: protein language models, unsupervised learning, protein evolution, protein structure prediction, secondary structure prediction, protein tertiary structure, remote homology
MeSH terms: Unsupervised Machine Learning, Amino Acids, Protein Conformation, Sequence Homology, Amino Acid, Sequence Analysis, Protein

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Abstract

Integrity signals

Claims the paper makes

Self-check

Cites

Cited by

Related

Record

About these checks