Evaluating Protein Transfer Learning with TAPE
Open accessAbstract
Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to real-life scenarios. We bench-mark a range of approaches to semi-supervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that self-supervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by self-supervised pretraining still lag behind features extracted by state-of-the-art non-neural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https://github.com/songlab-cal/tape .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Language models learn summary representations that can be used to capture sequence relationships and link sequence to function via transfer learning.
“They also learn summary representations, powerful features that can be used to better capture sequence relationships and link sequence to function via transfer learning 27,28,32-35.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [33]
Various language model-based protein embedding methods have been applied to protein prediction problems through transfer learning, including protein phenotype prediction, residue-residue contact prediction, fold recognition, protein-protein and protein-drug interaction prediction.
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669 · see citing works
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302 · see citing works
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Identification of common molecular subsequences
Smith, T. F., & Waterman, M. S. (1981) · Journal of Molecular Biology · doi:10.1016/0022-2836(81)90087-5 · Cited by 9,962 · see citing works
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698 · see citing works
The SWISS-MODEL workspace: a web-based environment for protein structure homology modelling
Arnold, K., Bordoli, L., Kopp, J., & Schwede, T. F. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti770 · Cited by 6,951 · see citing works
Eddy, S. R. (1998) · Bioinformatics · doi:10.1093/bioinformatics/14.9.755 · Cited by 5,515 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
The Pfam protein families database in 2019
El-Gebali, S., Mistry, J., Bateman, A., Eddy, S. R., Luciani, A., Potter, S. C., Qureshi, M., Richardson, L. J., Salazar, G. A., Smart, A., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky995 · Cited by 4,602 · see citing works
Chapelle, O., Schölkopf, B., & Zien, A. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262033589.001.0001 · Cited by 4,560 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
The HHpred interactive server for protein homology detection and structure prediction
Söding, J., Biegert, A., & Lupas, A. N. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki408 · Cited by 3,703 · see citing works
Biological sequence analysis probabilistic models of proteins and nucleic acids
Durbin, R. M., Eddy, S. R., Krogh, A., & Mitchison, G. (1998) · Cited by 3,565 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
Durbin, R. M., Eddy, S. R., Krogh, A., & Mitchison, G. (1998) · Cambridge University Press eBooks · doi:10.1017/cbo9780511790492 · Cited by 3,255 · see citing works
Proteins: Structures and Molecular Properties
Creighton, T. E. (1986) · Medical Entomology and Zoology · Cited by 2,652 · see citing works
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288 · see citing works
Yu, F., Koltun, V., & Funkhouser, T. A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.75 · Cited by 1,925 · see citing works
Twilight zone of protein sequence alignments
Rost, B. (1999) · Protein Engineering, Design and Selection · doi:10.1093/protein/12.2.85 · Cited by 1,833 · see citing works
JPred4: a protein secondary structure prediction server
Drozdetskiy, A., Cole, C., Procter, J. B., & Barton, G. J. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv332 · Cited by 1,714 · see citing works
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Černocký, J., & Khudanpur, S. P. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947611 · Cited by 1,594 · see citing works
A series of PDB related databases for everyday needs
Joosten, R. P., Beek, T. A. H. T., Krieger, E., Hekkelman, M. L., Hooft, R. W. W., Schneider, R., Sander, C., & Vriend, G. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq1105 · Cited by 1,298 · see citing works
THE KINETICS OF FORMATION OF NATIVE RIBONUCLEASE DURING OXIDATION OF THE REDUCED POLYPEPTIDE CHAIN
Anfinsen, C. B., Haber, E., Sela, M., & White, F. H. (1961) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.47.9.1309 · Cited by 1,182 · see citing works
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Fox, N. K., Brenner, S. E., & Chandonia, J.-M. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1240 · Cited by 762 · see citing works
Critical assessment of methods of protein structure prediction (CASP) — round x
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2013) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24452 · Cited by 673 · see citing works
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
Cuff, J. A., & Barton, G. J. (1999) · Proteins Structure Function and Bioinformatics · doi:10.1002/(sici)1097-0134(19990301)34:4 · Cited by 660 · see citing works
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · see citing works
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606 · see citing works
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601 · see citing works
Major New Microbial Groups Expand Diversity and Alter our Understanding of the Tree of Life
Castelle, C. J., & Banfield, J. F. (2018) · Cell · doi:10.1016/j.cell.2018.02.016 · Cited by 595 · see citing works
Local fitness landscape of the green fluorescent protein
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, D. R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N., Bozhanova, N. G., Baranov, M. S., Soylemez, O., et al. (2016) · Nature · doi:10.1038/nature17995 · Cited by 591 · see citing works
Brenner, S. E., Chothia, C., & Hubbard, T. (1998) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.95.11.6073 · Cited by 525 · see citing works
Global analysis of protein folding using massively parallel design, synthesis, and testing
Rocklin, G. J., Chidyausiku, T. M., Goreshnik, I., Ford, A., Houliston, S., Lemak, A., Carter, L., Ravichandran, R., Mulligan, V. K., Chevalier, A., et al. (2017) · Science · doi:10.1126/science.aan0693 · Cited by 501 · see citing works
CasX enzymes comprise a distinct family of RNA-guided genome editors
Liu, J.-J., Orlova, N., Oakes, B. L., Ma, E., Spinner, H. B., Baney, K. L. M., Chuck, J., Tan, D., Knott, G. J., Harrington, L. B., et al. (2019) · Nature · doi:10.1038/s41586-019-0908-x · Cited by 446 · see citing works
CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations
Seemayer, S., Gruber, M., & Söding, J. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu500 · Cited by 416 · see citing works
Whitehead, T. A., Chevalier, A. A., Song, Y., Dreyfus, C., Fleishman, S. J., de Mattos, C., Myers, C., Kamisetty, H., Blair, P. J., Wilson, I. A., et al. (2012) · Nature Biotechnology · doi:10.1038/nbt.2214 · Cited by 364 · see citing works
Self-citationCritical assessment of methods of protein structure prediction (CASP)—Round XII
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25415 · Cited by 327 · see citing works
Learned protein embeddings for machine learning
Yang, K., Wu, Z., Bedbrook, C. N., & Arnold, F. H. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty178 · Cited by 316 · see citing works
Sixty-five years of the long march in protein secondary structure prediction: the final stretch?
Yang, Y., Gao, J., Wang, J., Heffernan, R., Hanson, J., Paliwal, K. K., & Zhou, Y. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw129 · Cited by 237 · see citing works
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · see citing works
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219 · see citing works
Unexpected features of the dark proteome
Perdigão, N., Heinrich, J., Stolte, C., Sabir, K. S., Buckley, M. J., Tabor, B., Signal, B., Gloss, B., Hammang, C. J., Rost, B., et al. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1508380112 · Cited by 210 · see citing works
Assessment of contact predictions in CASP12: Co‐evolution and deep learning coming of age
Schaarschmidt, J., Monastyrskyy, B., Kryshtafovych, A., & Bonvin, A. M. J. J. (2018) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25407 · Cited by 177 · see citing works
Nomenclature and symbolism for amino acids and peptides (Recommendations 1983)
(1984) · Pure and Applied Chemistry · doi:10.1351/pac198456050595 · Cited by 143 · see citing works
ProteinNet: a standardized data set for machine learning of protein structure
AlQuraishi, M. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2932-0 · Cited by 141 · see citing works
Ma, J., Wang, S., Wang, Z., & Xu, J. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv472 · Cited by 107 · see citing works
Protein Structure Relationships Revealed by Mutational Analysis
Yanofsky, C., Horn, V., & Thorpe, D. (1964) · Science · doi:10.1126/science.146.3651.1593 · Cited by 98 · see citing works
Coordinated amino acid changes in homologous protein families
Altschuh, D., Vernet, T., Berti, P. J., Moras, D., & Nagai, K. (1988) · Protein Engineering Design and Selection · doi:10.1093/protein/2.3.193 · Cited by 96 · see citing works
Kim, D. E., DiMaio, F., Wang, R. Y., Song, Y., & Baker, D. A. (2013) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24374 · Cited by 90 · see citing works
Self-citationTavares, L. S., da Silva, C. G., de Souza, V. C., da Silva, V. L., Diniz, C. G., & Santos, M. D. O. (2013) · Frontiers in Microbiology · doi:10.3389/fmicb.2013.00412 · Cited by 73 · see citing works
Semi-Supervised Protein Classification Using Cluster Kernels
Weston, J., Leslie, C. S., Ie, E., & Noble, W. S. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/6173.003.0026 · Cited by 45 · see citing works
Modular Protein Engineering in Emerging Cancer Therapies
Vázquez, E., Ferrer‐Miralles, N., Mangues, R., Corchero, J. L., Schwartz, S., & Villaverde, A. (2009) · Current Pharmaceutical Design · doi:10.2174/138161209787582084 · Cited by 40 · see citing works
Prediction of Protein Function from Networks
Hyunjung, S., & Koji, T. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262033589.003.0020 · Cited by 21 · see citing works
The cost and value of three-dimensional protein structure
Stevens, R. C. (2003) · Cited by 11 · see citing works
DeepSF: deep convolutional neural network for mapping protein sequences to foldsPreprint
Hou, J., Adhikari, B., & Cheng, J. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01010 · Cited by 9 · see citing works
Machine learning in protein engineeringPreprint
Yang, K., Wu, Z., & Arnold, F. H. (2018) · arXiv (Cornell University) · Cited by 9 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 1
- Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training Deep Nets with Sublinear Memory Cost. arXiv, 2016.
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
A guide to machine learning for biologists
Greener, J. G., Kandathil, S. M., Moffat, L., & Jones, D. T. (2022) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-021-00407-0 · Cited by 2,127 · see citing works
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078 · see citing works
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · see citing works
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795 · see citing works
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635 · see citing works
Self-citationLearning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483 · see citing works
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations
Das, P., Sercu, T., Wadhawan, K., Padhi, I., Gehrmann, S., Cipcigan, F. S., Chenthamarakshan, V., Strobelt, H., dos Santos, C., Chen, P.-Y., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00689-x · Cited by 468 · see citing works
Efficient evolution of human antibodies from general protein language models
Hie, B. L., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P., Tang, S.-G., Wu, W., Pak, J., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 452 · see citing works
Learning inverse folding from millions of predicted structuresPreprint
Hsu, C., Verkuil, R., Liu, J. J., Lin, Z., Hie, B., Sercu, T., Lerer, A. K., & Rives, A. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 433 · see citing works
Genome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 398 · see citing works
Machine Learning in Enzyme Engineering
Mazurenko, S., Prokop, Z., & Damborský, J. (2019) · ACS Catalysis · doi:10.1021/acscatal.9b04321 · Cited by 410 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
“We use a deep bidirectional Transformer encoder model (Devlin et al., 2018; Vaswani et al., 2017) and process data at the character-level, corresponding to individual amino-acids in our application.”
MethodProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350 · see citing works
Self-citationLearning the language of viral evolution and escape
Hie, B., Zhong, E. D., Berger, B., & Bryson, B. (2021) · Science · doi:10.1126/science.abd7331 · Cited by 361 · see citing works
Deep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350 · see citing works
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338 · see citing works
The language of proteins: NLP, machine learning & protein sequences
Ofer, D., Brandes, N., & Linial, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.03.022 · Cited by 351 · see citing works
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
Self-citationProtein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320 · see citing works
Xie, J., Zhang, J., Sun, J., Ma, Z., Qin, L., Li, G., Zhou, H., & Zhan, Y. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3194600 · Cited by 323 · see citing works
Utilizing graph machine learning within drug discovery and development
Gaudelet, T., Day, B., Jamasb, A. R., Soman, J., Regep, C., Liu, G., Hayter, J. B. R., Vickers, R., Roberts, C., Tang, J., et al. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab159 · Cited by 313 · see citing works
flDPnn: Accurate intrinsic disorder prediction with putative propensities of disorder functions
Hu, G., Katuwawala, A., Wang, K., Wu, Z., Ghadermarzi, S., Gao, J., & Kurgan, L. (2021) · Nature Communications · doi:10.1038/s41467-021-24773-7 · Cited by 292 · see citing works
Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering
Yang, J., Li, F.-Z., & Arnold, F. H. (2024) · ACS Central Science · doi:10.1021/acscentsci.3c01275 · Cited by 223 · see citing works
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 251 · see citing works
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 239 · see citing works
Fine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229 · see citing works
Deep Learning in Protein Structural Modeling and Design
Gao, W., Mahajan, S. P., Sulam, J., & Gray, J. J. (2020) · Patterns · doi:10.1016/j.patter.2020.100142 · Cited by 227 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N., Anand, N., Eguchi, R. R., Huang, P.-S., & Socher, R. (2020) · bioRxiv · doi:10.1101/2020.03.07.982272 · Cited by 361 · see citing works
Informed training set design enables efficient machine learning-assisted directed protein evolution
Wittmann, B. J., Yue, Y., & Arnold, F. H. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.008 · Cited by 207 · see citing works
Generating functional protein variants with variational autoencoders
Hawkins‐Hooker, A., Depardieu, F., Baur, S., Couairon, G., Chen, A., & Bikard, D. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008736 · Cited by 211 · see citing works
Learning protein fitness models from evolutionary and assay-labeled data
Hsu, C., Nisonoff, H. M., Fannjiang, C., & Listgarten, J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01146-5 · Cited by 222 · see citing works
Learning functional properties of proteins with language models
Ünsal, S., Ataş, H., Albayrak, M., Turhan, K., Acar, A. C., & Doğan, T. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00457-9 · Cited by 197 · see citing works
Controllable protein design with language models
Ferruz, N., & Höcker, B. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00499-z · Cited by 196 · see citing works
ProteInfer, deep neural networks for protein functional inference
Sanderson, T., Bileschi, M. L., Belanger, D., & Colwell, L. J. (2023) · eLife · doi:10.7554/elife.80942 · Cited by 192 · see citing works
Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning
Kroll, A., Rousset, Y., Hu, X.-P., Liebrand, N. A., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-39840-4 · Cited by 186 · see citing works
Transformer-based deep learning for predicting protein properties in the life sciences
Chandra, A. A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 192 · see citing works
Jhong, J.-H., Yao, L., Pang, Y., Li, Z., Chung, C., Wang, R., Li, S., Li, W., Luo, M., Ma, R., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab1080 · Cited by 186 · see citing works
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 201 · see citing works
TANKBind: Trigonometry-Aware Neural NetworKs for Drug-Protein Binding Structure Prediction
Lu, W., Wu, Q., Zhang, J., Rao, J., Li, C., & Zheng, S. (2022) · neural information processing systems · doi:10.52202/068431-0525 · Cited by 174 · see citing works
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2023) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 206 · see citing works
ECNet is an evolutionary context-integrated deep learning framework for protein engineering
Luo, Y., Jiang, G., Yu, T., Liu, Y., Vo, L. T., Ding, H., Su, Y., Qian, W. W., Zhao, H., & Peng, J. (2021) · Nature Communications · doi:10.1038/s41467-021-25976-8 · Cited by 171 · see citing works
Learning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 163 · see citing works
Blockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 179 · see citing works
Advances in machine learning for directed evolution
Wittmann, B. J., Johnston, K. E., Wu, Z., & Arnold, F. H. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.01.008 · Cited by 165 · see citing works
AMPGAN v2: Machine Learning-Guided Design of Antimicrobial Peptides
Oort, C. M. V., Ferrell, J. B., Remington, J. M., Wshah, S., & Li, J. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01441 · Cited by 168 · see citing works
Deciphering the language of antibodies using self-supervised learning
Leem, J., Mitchell, L., Farmery, J. H. R., Barton, J., & Galson, J. D. (2022) · Patterns · doi:10.1016/j.patter.2022.100513 · Cited by 156 · see citing works
Hie, B., Yang, K., & Kim, P. S. (2022) · Cell Systems · doi:10.1016/j.cels.2022.01.003 · Cited by 147 · see citing works
Single-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150 · see citing works
Narayanan, H., Dingfelder, F., Butté, A., Lorenzen, N., Sokolov, M., & Arosio, P. (2021) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2020.12.004 · Cited by 146 · see citing works
Anticancer peptides prediction with deep representation learning features
Lv, Z., Cui, F., Zou, Q., Zhang, L., & Xu, L. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab008 · Cited by 145 · see citing works
Protein remote homology detection and structural alignment using deep learning
Hamamsy, T., Morton, J. T., Blackwell, R., Berenberg, D., Carriero, N. J., Gligorijević, V., Strauss, C. E. M., Leman, J. K., Cho, K., & Bonneau, R. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01917-2 · Cited by 145 · see citing works
Protein sequence design with deep generative models
Wu, Z., Johnston, K. E., Arnold, F. H., & Yang, K. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.004 · Cited by 144 · see citing works
Active learning-assisted directed evolution
Yang, J., Lal, R., Bowden, J. C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., & Arnold, F. H. (2025) · Nature Communications · doi:10.1038/s41467-025-55987-8 · Cited by 144 · see citing works
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139 · see citing works
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137 · see citing works
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130 · see citing works
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132 · see citing works
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 127 · see citing works
Self-citationScientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 155 · see citing works
Transformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 133 · see citing works
Functional annotation of enzyme-encoding genes using deep learning with transformer layers
Kim, G. B., Kim, J. Y., Lee, J.-A., Norsigian, C. J., Palsson, B., & Lee, S. Y. (2023) · Nature Communications · doi:10.1038/s41467-023-43216-z · Cited by 115 · see citing works
Machine learning modeling of family wide enzyme-substrate specificity screens
Goldman, S., Das, R., Yang, K. K., & Coley, C. W. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009853 · Cited by 120 · see citing works
Sparks of function by de novo protein design
Chu, A. E., Lu, T.-Y., & Huang, P.-S. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02133-2 · Cited by 116 · see citing works
Foundation models in bioinformatics
Guo, F., Guan, R., Li, Y., Liu, Q., Wang, X., Yang, C., & Wang, J. (2025) · National Science Review · doi:10.1093/nsr/nwaf028 · Cited by 120 · see citing works
A foundational large language model for edible plant genomes
Mendoza‐Revilla, J., Trop, E., Gonzalez, L., Roller, M., Dalla-Torre, H., de Almeida, B. P., Richard, G., Caton, J., Carranza, N. L., Skwark, M. J., et al. (2024) · Communications Biology · doi:10.1038/s42003-024-06465-2 · Cited by 114 · see citing works
A new age in protein design empowered by deep learning
Khakzad, H., Igashov, I., Schneuing, A., Goverde, C. A., Bronstein, M. M., & Correia, B. E. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.006 · Cited by 113 · see citing works
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113 · see citing works
“We use the datasets provided by TAPE (Rao et al. 2019; Fox, Brenner, and Chandonia 2013; Hou, Adhikari, and Cheng 2018), where the train/test split is generated by holding out certain evolutionary groups.”
BackgroundMethodSelf-citationxTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of ProteinPreprint
Chen, B., Cheng, X., Li, P., Geng, Y.-A., Gong, J., Li, S., Bei, Z.-L., Tan, X., Wang, B., Zeng, X., et al. (2024) · bioRxiv · doi:10.1101/2023.07.05.547496 · Cited by 157 · see citing works
Machine learning to navigate fitness landscapes for protein engineering
Freschlin, C. R., Fahlberg, S. A., & Romero, P. A. (2022) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2022.102713 · Cited by 105 · see citing works
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112 · see citing works
Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets
Dallago, C., Schütze, K., Heinzinger, M., Olenyi, T., Littmann, M., Lu, A. X., Yang, K., Min, S., Yoon, S., Morton, J. T., et al. (2021) · Current Protocols · doi:10.1002/cpz1.113 · Cited by 101 · see citing works
Mol‐BERT: An Effective Molecular Representation with BERT for Molecular Property Prediction
Li, J., & Jiang, X. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/7181815 · Cited by 112 · see citing works
Biological Sequence Classification: A Review on Data and General Methods
Ao, C., Jiao, S., Wang, Y., Yu, L., & Zou, Q. (2022) · Research · doi:10.34133/research.0011 · Cited by 99 · see citing works
PepNN: a deep attention model for the identification of peptide binding sites
Abdin, O., Nim, S., Wen, H., & Kim, P. M. (2022) · Communications Biology · doi:10.1038/s42003-022-03445-2 · Cited by 104 · see citing works
PLMSearch: Protein language model powers accurate and fast sequence search for remote homology
Liu, W., Wang, Z., You, R., Xie, C., Wei, H., Xiong, Y., Yang, J., & Zhu, S. (2024) · Nature Communications · doi:10.1038/s41467-024-46808-5 · Cited by 96 · see citing works
Li, L., Gupta, E., Spaeth, J., Shing, L., Jaimes, R., Engelhart, E., Lopez, R., Caceres, R. S., Bepler, T., & Walsh, M. E. (2023) · Nature Communications · doi:10.1038/s41467-023-39022-2 · Cited by 96 · see citing works
EEG Classification with Transformer-Based Models
Sun, J., Xie, J., & Zhou, H. (2021) · IEEE Global Conference on Life Sciences and Technologies (LifeTech) · doi:10.1109/lifetech52111.2021.9391844 · Cited by 93 · see citing works
A survey of drug-target interaction and affinity prediction methods via graph neural networks
Zhang, Y., Hu, Y., Han, N., Yang, A., Liu, X., & Cai, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107136 · Cited by 93 · see citing works
GEFA: Early Fusion Approach in Drug-Target Affinity Prediction
Nguyen, T. M., Nguyen, T., Le, T. M., & Tran, T. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3094217 · Cited by 91 · see citing works
Machine learning approaches to predict drug efficacy and toxicity in oncology
Badwan, B. A., Liaropoulos, G., Kyrodimos, E., Skaltsas, D., Tsirigos, A., & Gorgoulis, V. G. (2023) · Cell Reports Methods · doi:10.1016/j.crmeth.2023.100413 · Cited by 90 · see citing works
NetGO 3.0: Protein Language Model Improves Large-Scale Functional Annotations
Wang, S., You, R., Liu, Y., Xiong, Y., & Zhu, S. (2023) · Genomics Proteomics & Bioinformatics · doi:10.1016/j.gpb.2023.04.001 · Cited by 90 · see citing works
Qiu, Y., & Wei, G. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad289 · Cited by 90 · see citing works
McInnes, G., Sharo, A. G., Koleske, M. L., Brown, J. E. H., Norstad, M., Adhikari, A. N., Wang, S., Brenner, S. E., Halpern, J., Koenig, B. A., et al. (2021) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2021.03.003 · Cited by 86 · see citing works
“Transformer-based approaches, frequently used in natural language processing, have been applied to functional predictions of variants in proteins.(91,92) Graph-CNNs have been used to make drug-binding predictions with protein structure data after being pre-trained with an unsupervised learning step.”
MethodLindorff-Larsen, K., & Kragelund, B. B. (2021) · Journal of Molecular Biology · doi:10.1016/j.jmb.2021.167196 · Cited by 85 · see citing works
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C. J., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 84 · see citing works
Cheng, J., Bendjama, K., Rittner, K., & Malone, B. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab422 · Cited by 83 · see citing works
Improving protein succinylation sites prediction using embeddings from protein language model
Pokharel, S., Pratyush, P., Heinzinger, M., Newman, R. H., & KC, D. B. (2022) · Scientific Reports · doi:10.1038/s41598-022-21366-2 · Cited by 86 · see citing works
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 83 · see citing works
Perceiver CPI: a nested cross-attention network for compound–protein interaction prediction
Nguyen, N.-Q., Jang, G., Kim, H., & Kang, J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btac731 · Cited by 83 · see citing works
Enhancing scientific discoveries in molecular biology with deep generative models
Lopez, R., Gayoso, A., & Yosef, N. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199198 · Cited by 82 · see citing works
Ultrafast homomorphic encryption models enable secure outsourcing of genotype imputation
Kim, M., Harmanci, A. O., Bossuat, J.-P., Carpov, S., Cheon, J. H., Chillotti, I., Cho, W., Froelicher, D., Gama, N., Georgieva, M., et al. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.010 · Cited by 79 · see citing works
Self-citation
- Authors
- Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S.
- Affiliations
- University of California, Berkeley (US)
- Google (United States) (US)
- Corresponding author
- Rao, Roshan
- Bhattacharya, Nicholas
- Thomas, Neil
- Song, Yun S.
- ORCID
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-06-20
- OpenAlex
- W2951433247
- Cited by
- 719
- References
- 62
- Citations by year
- 2026: 55 · 2025: 114 · 2024: 150 · 2023: 148 · 2022: 118 · 2021: 100 · 2020: 31 · 2019: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- Funders
- Open Philanthropy Project
- National Institutes of Health
- Defense Advanced Research Projects Agency
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- protein representation learning, semi-supervised learning, protein embeddings, transfer learning, self-supervised pre-training, benchmark dataset, protein structure prediction, protein function prediction, sequence learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
