ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Open accessPublished version of a preprint
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Fine-tuning has not been extensively explored in previous work.
“Fine-tuning has not been extensively explored in previous work5, possibly due to its demanding computational requirements.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
The highest model performance is never achieved by using embeddings from the final layer.
“We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
The BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model for protein sequence prediction.
“Such models were then applied to protein sequence: the BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model [58] for protein sequence prediction.”
From Computational and artificial intelligence-based methods for antibody development (2023) · cites this paper as [58]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Amino acid substitution matrices from protein blocks.
Henikoff, S., & Henikoff, J. G. (1992) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.89.22.10915 · Cited by 6,218 · see citing works
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · see citing works
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Approximate nearest neighbors: towards removing the curse of dimensionality
Indyk, P., & Motwani, R. (1998) · Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 · doi:10.1145/276698.276876 · Cited by 4,188 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 4× · see citing works
“As previously established for another protein LM [24], the t-SNE projections (Fig.”
BackgroundMethodResultInfluentialTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Prediction of Protein Secondary Structure at Better than 70% Accuracy
Rost, B., & Sander, C. (1993) · Journal of Molecular Biology · doi:10.1006/jmbi.1993.1413 · Cited by 2,927 · see citing works
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · see citing works
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928 · see citing works
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844 · see citing works
JPred4: a protein secondary structure prediction server
Drozdetskiy, A., Cole, C., Procter, J. B., & Barton, G. J. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv332 · Cited by 1,714 · see citing works
PISCES: a protein sequence culling server
Wang, G., & Dunbrack, R. L. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg224 · Cited by 1,664 · see citing works
Question Answering For Toxicological Information Extraction
Ferreira, B. C. L., Gonçalo Oliveira, H., Amaro, H., et al. (2022) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.slate.2022.3 · Cited by 1,535 · see citing works
Combining evolutionary information and neural networks to predict protein secondary structure
Rost, B., & Sander, C. (1994) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.340190108 · Cited by 1,452 · see citing works
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · see citing works
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421 · see citing works
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works
[31] PHD: Predicting one-dimensional protein structure by profile-based neural networks
Rost, B. (1996) · Methods in enzymology on CD-ROM/Methods in enzymology · doi:10.1016/s0076-6879(96)66033-9 · Cited by 1,282 · see citing works
Bairoch, A. (2000) · Nucleic Acids Res. · doi:10.1093/nar/28.1.304 · Cited by 1,182 · see citing works
Protein 3D Structure Computed from Evolutionary Sequence Variation
Marks, D. S., Colwell, L. J., Sheridan, R., Hopf, T. A., Pagnani, A., Zecchina, R., & Sander, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028766 · Cited by 1,144 · see citing works
DeepLoc: prediction of protein subcellular localization using deep learning
Armenteros, J. J. A., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx431 · Cited by 1,144 · see citing works
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Clustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875 · see citing works
Horovod: fast and easy distributed deep learning in TensorFlowPreprint
Sergeev, A., & Balso, M. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05799 · Cited by 867 · see citing works
Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics
Asgari, E., & Mofrad, M. R. K. (2015) · PLoS ONE · doi:10.1371/journal.pone.0141287 · Cited by 817 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
Cuff, J. A., & Barton, G. J. (1999) · Proteins Structure Function and Bioinformatics · doi:10.1002/(sici)1097-0134(19990301)34:4 · Cited by 660 · see citing works
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627 · see citing works
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · see citing works
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606 · see citing works
Self-citationStudies on the Reduction and Re-formation of Protein Disulfide Bonds
Anfinsen, C. B., & Haber, E. (1961) · Journal of Biological Chemistry · doi:10.1016/s0021-9258(18)64177-8 · Cited by 603 · see citing works
A Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600 · see citing works
Large Batch Optimization for Deep Learning: Training BERT in 76 minutesPreprint
You, Y., Li, J., Reddi, S. J., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J. W., Keutzer, K., & Hsieh, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00962 · Cited by 597 · see citing works
NetSurfP-2.0: improved prediction of protein structural features by integrated deep learning
Klausen, M., Jespersen, M. C., Nielsen, H., Jensen, K. K., Jurtz, V., Sønderby, C., Sommer, M., Winther, O., Nielsen, M., Petersen, B., et al. (2018) · bioRxiv · doi:10.1002/prot.25674 · Cited by 564 · see citing works
Protein Secondary Structure Prediction Using Deep Convolutional Neural Fields
Wang, S., Peng, J., Ma, J., & Xu, J. (2016) · Scientific Reports · doi:10.1038/srep18962 · Cited by 556 · see citing works
RaptorX-Property: a web server for protein structure property prediction
Wang, S., Li, W., Liu, S., & Xu, J. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw306 · Cited by 553 · see citing works
Improved prediction of protein secondary structure by use of sequence profiles and neural networks.
Rost, B., & Sander, C. (1993) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.90.16.7558 · Cited by 516 · see citing works
Three-Dimensional Structures of Membrane Proteins from Genomic Sequencing
Hopf, T. A., Colwell, L. J., Sheridan, R., Rost, B., Sander, C., & Marks, D. S. (2012) · Cell · doi:10.1016/j.cell.2012.04.012 · Cited by 501 · see citing works
Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold
Steinegger, M., Mirdita, M., & Söding, J. (2019) · Nature Methods · doi:10.1038/s41592-019-0437-4 · Cited by 445 · see citing works
Self-citationNVIDIA cuda software and gpu parallel computing architecture
Kirk, D. B. (2007) · International Symposium on Memory Management · doi:10.1145/1296907.1296909 · Cited by 439 · see citing works
End-to-End Differentiable Learning of Protein Structure
AlQuraishi, M. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.006 · Cited by 371 · see citing works
The SWISS-PROT protein sequence data bank
Bairoch, A., & Boeckmann, B. (1991) · Nucleic Acids Research · doi:10.1093/nar/19.suppl.2247 · Cited by 362 · see citing works
Heffernan, R., Yang, Y., Paliwal, K. K., & Zhou, Y. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx218 · Cited by 358 · see citing works
Protein flexibility and intrinsic disorder
Predrag Radivojac, A., Zoran Obradovic, A., Smith, D. K., Zhu, G., Vučetić, S., Brown, C. J., Lawson, J. D., & Dunker, A. K. (2003) · Protein Science · doi:10.1110/ps.03128904 · Cited by 341 · see citing works
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 282 · see citing works
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 239 · see citing works
Sixty-five years of the long march in protein secondary structure prediction: the final stretch?
Yang, Y., Gao, J., Wang, J., Heffernan, R., Hanson, J., Paliwal, K. K., & Zhou, Y. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw129 · Cited by 237 · see citing works
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219 · see citing works
Unexpected features of the dark proteome
Perdigão, N., Heinrich, J., Stolte, C., Sabir, K. S., Buckley, M. J., Tabor, B., Signal, B., Gloss, B., Hammang, C. J., Rost, B., et al. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1508380112 · Cited by 210 · see citing works
Elrod-Erickson, M., Benson, T. E., & Pabo, C. O. (1998) · Structure · doi:10.1016/s0969-2126(98)00047-1 · Cited by 202 · see citing works
Embeddings from deep learning transfer GO annotations beyond homology
Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T., & Rost, B. (2021) · Scientific Reports · doi:10.1038/s41598-020-80786-0 · Cited by 169 · see citing works
Chandonia, J., Fox, N. K., & Brenner, S. E. (2018) · Nucleic Acids Research · doi:10.1093/nar/gky1134 · Cited by 167 · see citing works
Attention Interpretability Across NLP TasksPreprint
Vashishth, S., Upadhyay, S., Tomar, G. S., & Faruqui, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11218 · Cited by 160 · see citing works
Bridging the Protein Sequence-Structure Gap by Structure Predictions
Rost, B., & Sander, C. (1996) · Annual Review of Biophysics and Biomolecular Structure · doi:10.1146/annurev.bb.25.060196.000553 · Cited by 156 · see citing works
ProteinNet: a standardized data set for machine learning of protein structure
AlQuraishi, M. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2932-0 · Cited by 141 · see citing works
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130 · see citing works
PSI-2: Structural Genomics to Cover Protein Domain Family Space
Dessailly, B. H., Nair, R. P., Jaroszewski, L., Fajardo, J. E., Kouranov, A., Lee, D., Fiser, A., Godzik, A., Rost, B., & Orengo, C. A. (2009) · Structure · doi:10.1016/j.str.2009.03.015 · Cited by 129 · see citing works
LocTree2 predicts localization for all domains of life
Goldberg, T., Hamp, T., & Rost, B. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts390 · Cited by 114 · see citing works
Transforming the Language of Life
Nambiar, A., Heflin, M., Liu, S., Maslov, S., Hopkins, M., & Ritz, A. (2020) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3388440.3412467 · Cited by 97 · see citing works
Abriata, L. A., Tamò, G. E., Monastyrskyy, B., Kryshtafovych, A., & Peraro, M. D. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25423 · Cited by 90 · see citing works
Asgari, E., McHardy, A. C., & Mofrad, M. R. K. (2019) · Scientific Reports · doi:10.1038/s41598-019-38746-w · Cited by 77 · see citing works
ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.
Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2019) · Cited by 69 · see citing works
Enhanced protein domain discovery by using language modeling techniques from speech recognition
Coin, L., Bateman, A. G., & Durbin, R. M. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0737502100 · Cited by 60 · see citing works
TMSEG: Novel prediction of transmembrane helices
Bernhofer, M., Kloppmann, E., Reeb, J., & Rost, B. (2016) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25155 · Cited by 57 · see citing works
Accurate contact predictions using covariation techniques and machine learning
Kościółek, T., & Jones, D. T. (2015) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24863 · Cited by 56 · see citing works
Pre-Training of Deep Bidirectional Protein Sequence Representations With Structural Information
Min, S., Park, S., Kim, S., Choi, H.-S., Lee, B., & Yoon, S. (2021) · IEEE Access · doi:10.1109/access.2021.3110269 · Cited by 55 · see citing works
Pitfalls of protein sequence analysis
Rost, B., & Valencia, A. (1996) · Current Opinion in Biotechnology · doi:10.1016/s0958-1669(96)80124-8 · Cited by 44 · see citing works
Hydrophobic forces and the length limit of foldable protein domains
Lin, M. M., & Zewail, A. H. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1207382109 · Cited by 40 · see citing works
Pred‐MutHTP: Prediction of disease‐causing and neutral mutations in human transmembrane proteins
Kulandaisamy, A., Zaucha, J. M., Sakthivel, R., Frishman, D., & Gromiha, M. M. (2019) · Human Mutation · doi:10.1002/humu.23961 · Cited by 30 · see citing works
TFLMS: Large Model Support in TensorFlow by Graph RewritingPreprint
Le, T., Imai, H., Negishi, Y., & Kawachiya, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02037 · Cited by 27 · see citing works
Evolutionary couplings and sequence variation effect predict protein binding sites
Littmann, M., Hopf, T. A., & Rost, B. (2018) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25585 · Cited by 26 · see citing works
Announcing Supercomputer Summit
Wells, J. C., Bland, A. S. B., Nichols, J. A., Hack, J., Foertter, F., Hagen, G., Maier, T., Ashfaq, M., Messer, B., & Parete-Koon, S. T. (2016) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · Cited by 23 · see citing works
Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs
Feng, J., & Huang, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01127 · Cited by 20 · see citing works
Dark Proteins Important for Cellular Function
Schafferhans, A., O’Donoghue, S. I., Heinzinger, M., & Rost, B. (2018) · PROTEOMICS · doi:10.1002/pmic.201800227 · Cited by 9 · see citing works
Extreme Scale-out SuperMUC Phase 2 - lessons learnedPreprint
Hammer, N., Jamitzky, F., Satzger, H., Allalen, M., Block, A., Karmakar, A., Brehm, M., Bader, R., Iapichino, L., Ragagnin, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.01507 · Cited by 7 · see citing works
Optimal Gradient Checkpoint Search for Arbitrary Computation GraphsPreprint
Feng, J., & Huang, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00079 · Cited by 3 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
2 of the 101 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
“[53] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Debsindhu Bhowmik, and Burkhard Rost.”
BackgroundSignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01156-3 · Cited by 2,999 · see citing works
SignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000524414 · Cited by 1,905 · see citing works
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078 · see citing works
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · see citing works
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795 · see citing works
“Given both the growth of proteinsequence databases and the lack of annotation for a signi fi cant part of the protein space, protein sequences have become great candidates for unsupervised training 4,10,11 and now offer the opportunity to encode and generate protein sequences.”
BackgroundMethodSimulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937 · see citing works
DeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 723 · see citing works
“…transformer models, the 12-layer ESM (Evolutionary Scale Modelling, (21)) model with 84M parameters, the 33-layer ESM model with 650M parameters (23) and the 3B parameter ProtT5-XL-UniRef50 model (24), referred to as ESM12, ESM1b and ProtT5, respectively, throughout the rest of the manuscript.”
The STRING database in 2025: protein networks with directionality of regulation
Szklarczyk, D., Nastou, K. C., Koutrouli, M., Kirsch, R. L., Mehryary, F., Hachilif, R., Hu, D., Peluso, M. E., Huang, Q. D., Fang, T., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae1113 · Cited by 856 · see citing works
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 656 · see citing works
“Fine-tuning has not been extensively explored in previous work 5 , possibly due to its demanding computational requirements.”
InfluentialProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
Efficient evolution of human antibodies from general protein language models
Hie, B., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P. A. B., Tang, S., Wu, W. C. H., Pak, J. E., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 444 · see citing works
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455 · see citing works
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455 · see citing works
Learning inverse folding from millions of predicted structuresPreprint
Hsu, C., Verkuil, R., Liu, J. J., Lin, Z., Hie, B., Sercu, T., Lerer, A. K., & Rives, A. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 434 · see citing works
High-resolution <i>de novo</i> structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392 · see citing works
Høie, M. H., Kiehl, E. N., Petersen, B., Nielsen, M., Winther, O., Nielsen, H., Hallgren, J., & Marcatili, P. (2022) · Nucleic Acids Res. · doi:10.1093/nar/gkac439 · Cited by 325 · see citing works
“Recent breakthroughs in the field of natural language processing (NLP) (13) have allowed for generating information-rich representations of protein sequences without the explicit need for MSAs (14,15).”
BackgroundToxinPred 3.0: An improved method for predicting the toxicity of peptides
Rathore, A. S., Choudhury, S., Arora, A., Tijare, P. A., & Raghava, G. P. S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108926 · Cited by 351 · see citing works
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“In proteomics, protein language models are used for secondary structure and contact prediction (ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property…”
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 339 · see citing works
“Journal of molecular biology, 433(18):167113, 2021.”
BackgroundMachine learning for functional protein design
Notin, P., Rollins, N. J., Gal, Y., Sander, C., & Marks, D. S. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 278 · see citing works
Tubiana, J., Schneidman-Duhovny, D., & Wolfson, H. J. (2022) · Nature Methods · doi:10.1038/s41592-022-01490-7 · Cited by 274 · see citing works
Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering
Yang, J., Li, F.-Z., & Arnold, F. H. (2024) · ACS Central Science · doi:10.1021/acscentsci.3c01275 · Cited by 257 · see citing works
Bilingual language model for protein sequence and structure
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., & Rost, B. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae150 · Cited by 311 · see citing works
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 239 · see citing works
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · see citing works
“Elnaggar et al. [81] devised ProtT5-XXL (11B) which was first trained on BFD [82] and then fine-tuned on UniRef50 [83] to predict the secondary structure.”
MethodDeepLoc 2.1: multi-label membrane protein type prediction using protein language models
Ødum, M. T., Teufel, F., Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae237 · Cited by 230 · see citing works
UniKP: a unified framework for the prediction of enzyme kinetic parameters
Yu, H., Deng, H., He, J., Keasling, J. D., & Luo, X. (2023) · Nature Communications · doi:10.1038/s41467-023-44113-1 · Cited by 226 · see citing works
“We speculated that pretrained models have greatly contributed to the performanceofUniKPbycreatinganeasilylearnablerepresentationof enzyme sequences and substrate structures using unsupervised information from the entire database 18 .”
InfluentialFine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229 · see citing works
Rapid in silico directed evolution by a protein language model with EVOLVEpro.
Jiang, K.-Y., Yan, Z.-Q., Bernardo, M. D., Sgrizzi, S. R., Villiger, L., Kayabolen, A., Kim, B. J., Carscadden, J. K., Hiraizumi, M., Nishimasu, H., et al. (2024) · Science · doi:10.1126/science.adr6006 · Cited by 248 · see citing works
Machine Learning-Guided Protein Engineering
Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlář, J., Damborský, J., Pluskal, T., Šivic, J., & Mazurenko, S. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c02743 · Cited by 214 · see citing works
(2023) · doi:10.1101/2023.12.07.570727 · Cited by 207 · see citing works
“• VESPA VESPA [Marquet et al., 2022] is as Single Amino acid Variant (SAV) effect predictor based on a combination the embeddings from the protein language model ProtT5 [Elnaggar et al., 2021], as well as per-residue conservation predictions.”
MethodAbLang: an antibody language model for completing antibody sequences
Olsen, T. H., Moal, I. H., & Deane, C. M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac046 · Cited by 198 · see citing works
Controllable protein design with language models
Ferruz, N., & Höcker, B. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00499-z · Cited by 197 · see citing works
ProteInfer, deep neural networks for protein functional inference
Sanderson, T., Bileschi, M. L., Belanger, D., & Colwell, L. J. (2023) · eLife · doi:10.7554/elife.80942 · Cited by 192 · see citing works
A general model to predict small molecule substrates of enzymes based on machine and deep learning
Kroll, A., Ranjan, S., Engqvist, M. K. M., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38347-2 · Cited by 193 · see citing works
“In order to create a single fi xed-length numerical representation of the whole protein, one typically calculates the element-wise mean across all amino acid representations 34,35,49 .”
BackgroundTransformer-based deep learning for predicting protein properties in the life sciences
Chandra, A. A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 192 · see citing works
“…tasks of secondary structure prediction ( Elnaggar et al., 2020a ; Rives et al., 2021 ; Brandes et al., 2021 ; Rao et al., 2021 ; Rao et al., 2019 ; Elnaggar et al., 2020b ; Sturmfels et al., 2020 ) and contact prediction Rives et al., 2021 ; Rao et al., 2021 ; He et al., 2021 ; Sturmfels et al.,…”
BackgroundMethodInfluentialApplications of transformer-based language models in bioinformatics: a survey
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176 · see citing works
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181 · see citing works
TITAN: T-cell receptor specificity prediction with bimodal attention networks
Weber, A., Born, J., & Martínez, M. R. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab294 · Cited by 169 · see citing works
Rapid protein stability prediction using deep learning representations
Blaabjerg, L. M., Kassem, M. M., Good, L. L., Jonsson, N., Cagiada, M., Johansson, K. E., Boomsma, W., Stein, A., & Lindorff-Larsen, K. (2023) · eLife · doi:10.7554/elife.82593 · Cited by 177 · see citing works
Wang, R., Jiang, Y., Jin, J., Yin, C., Yu, H., Wang, F., Feng, J., Su, R., Nakai, K., Zou, Q., et al. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad055 · Cited by 177 · see citing works
“RoutingTransformer (36) DNABERT, RNABERT, ProtBERT (5,17)”
MethodComputational and artificial intelligence-based methods for antibody development
Kim, J., McFee, M., Fang, Q., Abdin, O., & Kim, P. M. (2023) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2022.12.005 · Cited by 178 · see citing works
ECNet is an evolutionary context-integrated deep learning framework for protein engineering
Luo, Y., Jiang, G., Yu, T., Liu, Y., Vo, L. T., Ding, H., Su, Y., Qian, W. W., Zhao, H., & Peng, J. (2021) · Nature Communications · doi:10.1038/s41467-021-25976-8 · Cited by 171 · see citing works
Learning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 163 · see citing works
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 165 · see citing works
Protein generation with evolutionary diffusion: sequence is all you need
Alamdari, S., Thakkar, N., van den Berg, R., Tenenholtz, N., Strome, R., Moses, A. M., Lu, A. X., Fusi, N., Amini, A. P., & Yang, K. K. (2023) · doi:10.1101/2023.09.11.556673 · Cited by 154 · see citing works
MOFormer: Self-Supervised Transformer Model for Metal–Organic Framework Property Prediction
Cao, Z., Magar, R., Wang, Y., & Farimani, A. B. (2023) · Journal of the American Chemical Society · doi:10.1021/jacs.2c11420 · Cited by 167 · see citing works
Deciphering the language of antibodies using self-supervised learning
Leem, J., Mitchell, L., Farmery, J. H. R., Barton, J., & Galson, J. D. (2022) · Patterns · doi:10.1016/j.patter.2022.100513 · Cited by 154 · see citing works
“To visualize the AntiBERTa embeddings, 1,000 BCR heavy chainswere randomly selected from a well-characterized public dataset(40) and then averaged over the length dimension before projection by uniform manifold approximation and projection (UMAP).(26,29,41) Despite only being given BCR sequences and no other informati…”
BackgroundMethodInfluentialSingle-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150 · see citing works
Guiding questions to avoid data leakage in biological machine learning applications
Bernett, J., Blumenthal, D., Grimm, D. G., Haselbeck, F., Joeres, R., Kalinina, O. V., & List, M. (2024) · Nature Methods · doi:10.1038/s41592-024-02362-y · Cited by 155 · see citing works
Protein remote homology detection and structural alignment using deep learning
Hamamsy, T., Morton, J. T., Blackwell, R., Berenberg, D., Carriero, N. J., Gligorijević, V., Strauss, C. E. M., Leman, J. K., Cho, K., & Bonneau, R. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01917-2 · Cited by 145 · see citing works
Active learning-assisted directed evolution
Yang, J., Lal, R., Bowden, J. C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., & Arnold, F. H. (2025) · Nature Communications · doi:10.1038/s41467-025-55987-8 · Cited by 144 · see citing works
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139 · see citing works
“…using no other annotation than the amino acid sequences of 10^7-10^9 proteins (Alley et al., 2019; Asgari & Mofrad, 2015; Bepler & Berger, 2019, 2021; Elnaggar et al., 2021; Heinzinger et al., 2019; Madani et al., 2020; Ofer et al., 2021; Rao et al., 2019; Rives et al., 2021; Wu et al., 2021).”
BackgroundMethodInfluentialSelf-citationLight attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130 · see citing works
Yan, J., Cai, J., Zhang, B., Wang, Y., Wong, D. F., & Siu, S. W. I. (2022) · Antibiotics · doi:10.3390/antibiotics11101451 · Cited by 132 · see citing works
Rosen, Y., Brbić, M., Roohani, Y., Swanson, K., Li, Z., & Leskovec, J. (2024) · Nature Methods · doi:10.1038/s41592-024-02191-z · Cited by 125 · see citing works
RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks
Penić, R. J., Vlašić, T., Huber, R. G., Wan, Y., & Šikić, M. (2025) · Nature Communications · doi:10.1038/s41467-025-60872-5 · Cited by 146 · see citing works
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 127 · see citing works
“Recent efforts in sequence-function prediction [15, 16] have sought to leverage the information in these unlabeled sequences through pretraining and fine-tuning, and have successfully engineered proteins with brighter fluorescence and high catalytic efficiency [17].”
BackgroundScientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 155 · see citing works
Protein function prediction as approximate semantic entailment
Kulmanov, M., Guzmán-Vega, F. J., Duek Roggli, P., Lane, L., Arold, S. T., & Hoehndorf, R. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00795-w · Cited by 131 · see citing works
From sequence to function through structure: Deep learning for protein design
(2022) · doi:10.1016/j.csbj.2022.11.014 · Cited by 124 · see citing works
“ProtTrans (ProtElectra -Generator) [22] Optional: sequence Sequence Transformer yes UniRef50 420M 20/07”
BackgroundMethodInfluentialTransformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 133 · see citing works
Functional annotation of enzyme-encoding genes using deep learning with transformer layers
Kim, G. B., Kim, J. Y., Lee, J.-A., Norsigian, C. J., Palsson, B., & Lee, S. Y. (2023) · Nature Communications · doi:10.1038/s41467-023-43216-z · Cited by 115 · see citing works
Predictomes, a classifier-curated database of AlphaFold-modeled protein-protein interactions
Schmid, E. W., & Walter, J. C. (2025) · Molecular Cell · doi:10.1016/j.molcel.2025.01.034 · Cited by 120 · see citing works
“…pair that were external to AlphaFold, including co-dependency data from DEPMAP 46 , coexpression data 47 , T5 protein language model embeddings 48 , subcellular colocalization predictions from DeepLoc 2.0 49 , hit profiles from the BioORCS CRISPR screen database, and high throughput…”
InfluentialGenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231201154 · Cited by 120 · see citing works
Self-citationGenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2022) · doi:10.1101/2022.10.10.511571 · Cited by 120 · see citing works
Self-citationFoundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 130 · see citing works
Robust enzyme discovery and engineering with deep learning using CataPro
Wang, Z.-C., Xie, D., Wu, D., Luo, X.-Z., Wang, S., Li, Y., Yang, Y.-M., Li, W.-F., & Zheng, L.-Z. (2025) · Nature Communications · doi:10.1038/s41467-025-58038-4 · Cited by 128 · see citing works
PeptideBERT: A Language Model Based on Transformers for Peptide Property Prediction
Guntuboina, C., Das, A., Mollaei, P., Kim, S. W., & Farimani, A. B. (2023) · The Journal of Physical Chemistry Letters · doi:10.1021/acs.jpclett.3c02398 · Cited by 124 · see citing works
A survey of generative AI for de novo drug design: new frontiers in molecule and protein generation
Tang, X., Dai, H., Knight, E., Wu, F., Li, Y., Li, T., & Gerstein, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae338 · Cited by 126 · see citing works
“ProteinVAE [118] utilizes ProtBERT [119] to reduce raw input sequences into latent representations, employing an encoder–decoder framework with position-wise multi-head self-attention to capture long-range dependencies in these sequences.”
Foundation models in bioinformatics
Guo, F., Guan, R., Li, Y., Liu, Q., Wang, X., Yang, C., & Wang, J. (2025) · National Science Review · doi:10.1093/nsr/nwaf028 · Cited by 121 · see citing works
CatPred: a comprehensive framework for deep learning in vitro enzyme kinetic parameters
Boorla, V. S., & Maranas, C. D. (2025) · Nature Communications · doi:10.1038/s41467-025-57215-9 · Cited by 114 · see citing works
Contrastive learning on protein embeddings enlightens midnight zone
(2022) · doi:10.1093/nargab/lqac043 · Cited by 105 · see citing works
“As recommended 89 in the original publication (9), for ProtT5, we only used the encoder part 90 of ProtT5 in half-precision to embed protein sequences.”
BackgroundMethodResultInfluentialTMbed: transmembrane proteins predicted through language model embeddings
Bernhofer, M., & Rost, B. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04873-x · Cited by 105 · see citing works
“We generated embeddings with protein Language Models (pLMs) for our data sets using a transformer-based pLM ProtT5-XL-U50 (short: ProtT5) [34].”
BackgroundMethodInfluentialGhafarollahi, A., & Buehler, M. J. (2024) · Digital Discovery · doi:10.1039/d4dd00013g · Cited by 122 · see citing works
AlphaFold2 reveals commonalities and novelties in protein structure space for 21 model organisms
Bordin, N., Sillitoe, I., Nallapareddy, V., Rauer, C., Lam, S. D., Waman, V. P., Sen, N., Heinzinger, M., Littmann, M., Kim, S., et al. (2023) · Communications Biology · doi:10.1038/s42003-023-04488-9 · Cited by 103 · see citing works
“To determine whether these domains were distant homologues of CATH superfamilies, we used a recently established protein Language Model (pLM, Prot-T5 (24)) to generate protein sequence embeddings of the domains.”
MethodInfluentialSelf-citationAI-Driven Antimicrobial Peptide Discovery: Mining and Generation
Szymczak, P., Zarzecki, W., Wang, J., Duan, Y., Wang, J., Coelho, L. P., de la Fuente‐Núñez, C., & Szczurek, E. (2025) · Accounts of Chemical Research · doi:10.1021/acs.accounts.0c00594 · Cited by 113 · see citing works
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112 · see citing works
BioSeq-Diabolo: Biological sequence similarity analysis using Diabolo
Li, H., & Liu, B. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011214 · Cited by 103 · see citing works
A Transformer-Based Ensemble Framework for the Prediction of Protein–Protein Interaction Sites
Mou, M., Pan, Z., Zhou, Z., Zheng, L., Zhang, H., Shi, S., Li, F., Sun, X., & Zhu, F. (2023) · Research · doi:10.34133/research.0240 · Cited by 100 · see citing works
Fang, Y., Jiang, Y. F., Wei, L., Ma, Q., Ren, Z., Yuan, Q., & Wei, D. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad718 · Cited by 99 · see citing works
PepNN: a deep attention model for the identification of peptide binding sites
Abdin, O., Nim, S., Wen, H., & Kim, P. M. (2022) · Communications Biology · doi:10.1038/s42003-022-03445-2 · Cited by 104 · see citing works
“In addition, the idea of pre-training contextualized language models has recently been adapted to protein biology for the purpose of generating meaningful representations of protein sequences 23,24 .”
MethodInfluentialDeepSTABp: A Deep Learning Approach for the Prediction of Thermal Protein Stability
Jung, F., Frey, K. L., Zimmer, D., & Mühlhaus, T. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24087444 · Cited by 98 · see citing works
“It has been shown that the size of the training dataset correlates with predictive power of the language model effectively learning structure-relevant features that would be not possible to extract from the TPP dataset alone according to size limitations [29].”
BackgroundWang, L., Patena, W., Baalen, K. A. V., Xie, Y., Singer, E. R., Gavrilenko, S., Warren-Williams, M., Han, L., Harrigan, H. R., Hartz, L. D., et al. (2023) · Cell · doi:10.1016/j.cell.2023.06.008 · Cited by 99 · see citing works
PLMSearch: Protein language model powers accurate and fast sequence search for remote homology
Liu, W., Wang, Z., You, R., Xie, C., Wei, H., Xiong, Y., Yang, J., & Zhu, S. (2024) · Nature Communications · doi:10.1038/s41467-024-46808-5 · Cited by 96 · see citing works
SignalP: The Evolution of a Web Server
Nielsen, H., Teufel, F., Brunak, S., & von HEIJNE, G. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4007-4_17 · Cited by 100 · see citing works
AlQuraishi, M., & Sorger, P. K. (2021) · Nature Methods · doi:10.1038/s41592-021-01283-4 · Cited by 96 · see citing works
Genomic language model predicts protein co-regulation and function
Hwang, Y., Cornman, A. L., Kellogg, E. H., Ovchinnikov, S., & Girguis, P. R. (2024) · Nature Communications · doi:10.1038/s41467-024-46947-9 · Cited by 99 · see citing works
Yuan, Q., Xie, J., Xie, J., Zhao, H., & Yang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad117 · Cited by 95 · see citing works
Deep learning for the PSIPRED Protein Analysis Workbench
Buchan, D. W. A., Moffat, L., Lau, A. M. C., Kandathil, S. M., & Jones, D. T. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae328 · Cited by 94 · see citing works
Graph Neural Networks in Modern AI-Aided Drug Discovery
Zhang, O., Lin, H., Zhang, X., Wang, X., Wu, Z., Ye, Q., Zhao, W., Wang, J., Ying, K., Kang, Y., et al. (2025) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00461 · Cited by 103 · see citing works
Large language models improve annotation of prokaryotic viral proteins
Flamholz, Z. N., Biller, S. J., & Kelly, L. (2024) · Nature Microbiology · doi:10.1038/s41564-023-01584-8 · Cited by 92 · see citing works
“The Transformer_BFD PLM from the ProtTrans 28 project was used via the DeepChainBio/BioTransformers Python package (https://github. com/DeepChainBio/bio-transformers).”
Rules and mechanisms governing G protein coupling selectivity of GPCRs
Masuho, I., Kise, R., Gaínza, P., Moo, E. V., Li, X., Tany, R., Wakasugi-Masuho, H., Correia, B. E., & Martemyanov, K. A. (2023) · Cell Reports · doi:10.1016/j.celrep.2023.113173 · Cited by 91 · see citing works
“The pretrained protein BERT model from the ProtTrans suite 68 was installed using the Transformers library (https://huggingface.co/docs/transformers/index).”
Ni, B., Kaplan, D. L., & Buehler, M. J. (2023) · Chem · doi:10.1016/j.chempr.2023.03.020 · Cited by 87 · see citing works
Explainable Deep Hypergraph Learning Modeling the Peptide Secondary Structure Prediction
Jiang, Y., Wang, R., Feng, J., Jin, J., Liang, S., Li, Z., Yu, Y., Ma, A., Su, R., Zou, Q., et al. (2023) · Advanced Science · doi:10.1002/advs.202206151 · Cited by 89 · see citing works
Liu, S., Rodriguez, J. S., Munteanu, V., Ronkowski, C., Sharma, N. K., Alser, M., Andreace, F., Blekhman, R., Błaszczyk, D., Chikhi, R., et al. (2025) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00376-6 · Cited by 92 · see citing works
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Jiangnan University (CN)
- Google (United States) (US)
- Nvidia (United States) (US)
- NVIDIA, Santa Clara, CA, USA
- Seoul National University (KR)
- Oak Ridge National Laboratory (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 7112-7127
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3177500196
- Cited by
- 2,329
- References
- 101
- Citations by year
- 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100 · 2021: 25
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
- New Faculty Startup Fund and the Creative-Pioneering Researchers Program
- Seoul National University
- Oak Ridge National Laboratory
- U.S. Department of Energy (DE-AC05-00OR22725)
- TensorFlow Research Cloud
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
- MeSH terms
- Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
