The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Open accessPublished version of a preprint
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Fine-tuning has not been extensively explored in previous work5, possibly due to its demanding computational requirements.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“Such models were then applied to protein sequence: the BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model [58] for protein sequence prediction.”
From Computational and artificial intelligence-based methods for antibody development (2023) · cites this paper as [58]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Flagged first, then by citations
2 of the 101 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
- limited, 2019, Press release announcing Supercomputer Fugaku
- rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
- armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
- 10.1110/ps.03128904 · DOI
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
“[53] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Debsindhu Bhowmik, and Burkhard Rost.”
BackgroundSignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01156-3 · Cited by 3,088
SignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000524414 · Cited by 1,905
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
“Given both the growth of proteinsequence databases and the lack of annotation for a signi fi cant part of the protein space, protein sequences have become great candidates for unsupervised training 4,10,11 and now offer the opportunity to encode and generate protein sequences.”
BackgroundMethodSimulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937
DeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 736
“…transformer models, the 12-layer ESM (Evolutionary Scale Modelling, (21)) model with 84M parameters, the 33-layer ESM model with 650M parameters (23) and the 3B parameter ProtT5-XL-UniRef50 model (24), referred to as ESM12, ESM1b and ProtT5, respectively, throughout the rest of the manuscript.”
The STRING database in 2025: protein networks with directionality of regulation
Szklarczyk, D., Nastou, K. C., Koutrouli, M., Kirsch, R. L., Mehryary, F., Hachilif, R., Hu, D., Peluso, M. E., Huang, Q. D., Fang, T., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae1113 · Cited by 856
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 772
“Fine-tuning has not been extensively explored in previous work 5 , possibly due to its demanding computational requirements.”
InfluentialProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480
Efficient evolution of human antibodies from general protein language models
Hie, B., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P. A. B., Tang, S., Wu, W. C. H., Pak, J. E., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 472
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
Learning inverse folding from millions of predicted structuresPreprint
Hsu, C., Verkuil, R., Liu, J. J., Lin, Z., Hie, B., Sercu, T., Lerer, A. K., & Rives, A. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 492
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Høie, M. H., Kiehl, E. N., Petersen, B., Nielsen, M., Winther, O., Nielsen, H., Hallgren, J., & Marcatili, P. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac439 · Cited by 374
“Recent breakthroughs in the field of natural language processing (NLP) (13) have allowed for generating information-rich representations of protein sequences without the explicit need for MSAs (14,15).”
BackgroundToxinPred 3.0: An improved method for predicting the toxicity of peptides
Rathore, A. S., Choudhury, S., Arora, A., Tijare, P. A., & Raghava, G. P. S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108926 · Cited by 365
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“In proteomics, protein language models are used for secondary structure and contact prediction (ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property…”
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 364
“Journal of molecular biology, 433(18):167113, 2021.”
BackgroundMachine learning for functional protein design
Notin, P., Rollins, N. J., Gal, Y., Sander, C., & Marks, D. S. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 304
Tubiana, J., Schneidman-Duhovny, D., & Wolfson, H. J. (2022) · Nature Methods · doi:10.1038/s41592-022-01490-7 · Cited by 274
Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering
Yang, J., Li, F.-Z., & Arnold, F. H. (2024) · ACS Central Science · doi:10.1021/acscentsci.3c01275 · Cited by 268
Bilingual language model for protein sequence and structure
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., & Rost, B. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae150 · Cited by 311
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 246
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 267
“Elnaggar et al. [81] devised ProtT5-XXL (11B) which was first trained on BFD [82] and then fine-tuned on UniRef50 [83] to predict the secondary structure.”
MethodDeepLoc 2.1: multi-label membrane protein type prediction using protein language models
Ødum, M. T., Teufel, F., Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae237 · Cited by 230
UniKP: a unified framework for the prediction of enzyme kinetic parameters
Yu, H., Deng, H., He, J., Keasling, J. D., & Luo, X. (2023) · Nature Communications · doi:10.1038/s41467-023-44113-1 · Cited by 226
“We speculated that pretrained models have greatly contributed to the performanceofUniKPbycreatinganeasilylearnablerepresentationof enzyme sequences and substrate structures using unsupervised information from the entire database 18 .”
InfluentialFine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229
Rapid in silico directed evolution by a protein language model with EVOLVEpro
Jiang, K., Yan, Z., Bernardo, M. D., Sgrizzi, S. R., Villiger, L., Kayabölen, A., Kim, B., Carscadden, J. K., Hiraizumi, M., Nishimasu, H., et al. (2024) · Science · doi:10.1126/science.adr6006 · Cited by 234
Machine Learning-Guided Protein Engineering
Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlář, J., Damborský, J., Pluskal, T., Šivic, J., & Mazurenko, S. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c02743 · Cited by 218
(2023) · doi:10.1101/2023.12.07.570727 · Cited by 207
“• VESPA VESPA [Marquet et al., 2022] is as Single Amino acid Variant (SAV) effect predictor based on a combination the embeddings from the protein language model ProtT5 [Elnaggar et al., 2021], as well as per-residue conservation predictions.”
MethodAbLang: an antibody language model for completing antibody sequences
Olsen, T. H., Moal, I. H., & Deane, C. M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac046 · Cited by 198
Controllable protein design with language models
Ferruz, N., & Höcker, B. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00499-z · Cited by 209
ProteInfer, deep neural networks for protein functional inference
Sanderson, T., Bileschi, M. L., Belanger, D., & Colwell, L. J. (2023) · eLife · doi:10.7554/elife.80942 · Cited by 197
A general model to predict small molecule substrates of enzymes based on machine and deep learning
Kroll, A., Ranjan, S., Engqvist, M. K. M., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38347-2 · Cited by 193
“In order to create a single fi xed-length numerical representation of the whole protein, one typically calculates the element-wise mean across all amino acid representations 34,35,49 .”
BackgroundTransformer-based deep learning for predicting protein properties in the life sciences
Chandra, A. A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 202
“…tasks of secondary structure prediction ( Elnaggar et al., 2020a ; Rives et al., 2021 ; Brandes et al., 2021 ; Rao et al., 2021 ; Rao et al., 2019 ; Elnaggar et al., 2020b ; Sturmfels et al., 2020 ) and contact prediction Rives et al., 2021 ; Rao et al., 2021 ; He et al., 2021 ; Sturmfels et al.,…”
BackgroundMethodInfluentialApplications of transformer-based language models in bioinformatics: a survey
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181
TITAN: T-cell receptor specificity prediction with bimodal attention networks
Weber, A., Born, J., & Martínez, M. R. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab294 · Cited by 180
Rapid protein stability prediction using deep learning representations
Blaabjerg, L. M., Kassem, M. M., Good, L. L., Jonsson, N., Cagiada, M., Johansson, K. E., Boomsma, W., Stein, A., & Lindorff-Larsen, K. (2023) · eLife · doi:10.7554/elife.82593 · Cited by 177
Wang, R., Jiang, Y., Jin, J., Yin, C., Yu, H., Wang, F., Feng, J., Su, R., Nakai, K., Zou, Q., et al. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad055 · Cited by 178
“RoutingTransformer (36) DNABERT, RNABERT, ProtBERT (5,17)”
MethodComputational and artificial intelligence-based methods for antibody development
Kim, J., McFee, M., Fang, Q., Abdin, O., & Kim, P. M. (2023) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2022.12.005 · Cited by 178
ECNet is an evolutionary context-integrated deep learning framework for protein engineering
Luo, Y., Jiang, G., Yu, T., Liu, Y., Vo, L. T., Ding, H., Su, Y., Qian, W. W., Zhao, H., & Peng, J. (2021) · Nature Communications · doi:10.1038/s41467-021-25976-8 · Cited by 174
Learning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 171
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 191
Protein generation with evolutionary diffusion: sequence is all you need
Alamdari, S., Thakkar, N., van den Berg, R., Tenenholtz, N., Strome, R., Moses, A. M., Lu, A. X., Fusi, N., Amini, A. P., & Yang, K. K. (2023) · doi:10.1101/2023.09.11.556673 · Cited by 154
MOFormer: Self-Supervised Transformer Model for Metal–Organic Framework Property Prediction
Cao, Z., Magar, R., Wang, Y., & Farimani, A. B. (2023) · Journal of the American Chemical Society · doi:10.1021/jacs.2c11420 · Cited by 176
Deciphering the language of antibodies using self-supervised learning
Leem, J., Mitchell, L., Farmery, J. H. R., Barton, J., & Galson, J. D. (2022) · Patterns · doi:10.1016/j.patter.2022.100513 · Cited by 169
“To visualize the AntiBERTa embeddings, 1,000 BCR heavy chainswere randomly selected from a well-characterized public dataset(40) and then averaged over the length dimension before projection by uniform manifold approximation and projection (UMAP).(26,29,41) Despite only being given BCR sequences and no other informati…”
BackgroundMethodInfluentialSingle-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150
Guiding questions to avoid data leakage in biological machine learning applications
Bernett, J., Blumenthal, D. B., Grimm, D. G., Haselbeck, F., Joeres, R., Kalinina, O. V., & List, M. (2024) · Nature Methods · doi:10.1038/s41592-024-02362-y · Cited by 184
Protein remote homology detection and structural alignment using deep learning
Hamamsy, T., Morton, J. T., Blackwell, R., Berenberg, D., Carriero, N. J., Gligorijević, V., Strauss, C. E. M., Leman, J. K., Cho, K., & Bonneau, R. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01917-2 · Cited by 157
Active learning-assisted directed evolution
Yang, J., Lal, R., Bowden, J. C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., & Arnold, F. H. (2025) · Nature Communications · doi:10.1038/s41467-025-55987-8 · Cited by 144
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139
“…using no other annotation than the amino acid sequences of 10^7-10^9 proteins (Alley et al., 2019; Asgari & Mofrad, 2015; Bepler & Berger, 2019, 2021; Elnaggar et al., 2021; Heinzinger et al., 2019; Madani et al., 2020; Ofer et al., 2021; Rao et al., 2019; Rives et al., 2021; Wu et al., 2021).”
BackgroundMethodInfluentialSelf-citationLight attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130
Yan, J., Cai, J., Zhang, B., Wang, Y., Wong, D. F., & Siu, S. W. I. (2022) · Antibiotics · doi:10.3390/antibiotics11101451 · Cited by 138
Rosen, Y., Brbić, M., Roohani, Y., Swanson, K., Li, Z., & Leskovec, J. (2024) · Nature Methods · doi:10.1038/s41592-024-02191-z · Cited by 144
RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks
Penić, R. J., Vlašić, T., Huber, R. G., Wan, Y., & Šikić, M. (2025) · Nature Communications · doi:10.1038/s41467-025-60872-5 · Cited by 169
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 142
“Recent efforts in sequence-function prediction [15, 16] have sought to leverage the information in these unlabeled sequences through pretraining and fine-tuning, and have successfully engineered proteins with brighter fluorescence and high catalytic efficiency [17].”
BackgroundScientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 216
Protein function prediction as approximate semantic entailment
Kulmanov, M., Guzmán-Vega, F. J., Duek Roggli, P., Lane, L., Arold, S. T., & Hoehndorf, R. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00795-w · Cited by 131
From sequence to function through structure: Deep learning for protein design
(2022) · doi:10.1016/j.csbj.2022.11.014 · Cited by 124
“ProtTrans (ProtElectra -Generator) [22] Optional: sequence Sequence Transformer yes UniRef50 420M 20/07”
BackgroundMethodInfluentialTransformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 146
Functional annotation of enzyme-encoding genes using deep learning with transformer layers
Kim, G. B., Kim, J. Y., Lee, J. A., Norsigian, C. J., Palsson, B. Ø., & Lee, S. Y. (2023) · Nature Communications · doi:10.1038/s41467-023-43216-z · Cited by 122
Predictomes, a classifier-curated database of AlphaFold-modeled protein-protein interactions
Schmid, E. W., & Walter, J. C. (2025) · Molecular Cell · doi:10.1016/j.molcel.2025.01.034 · Cited by 120
“…pair that were external to AlphaFold, including co-dependency data from DEPMAP 46 , coexpression data 47 , T5 protein language model embeddings 48 , subcellular colocalization predictions from DeepLoc 2.0 49 , hit profiles from the BioORCS CRISPR screen database, and high throughput…”
InfluentialGenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231201154 · Cited by 120
Self-citationGenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2022) · doi:10.1101/2022.10.10.511571 · Cited by 120
Self-citationFoundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 162
Robust enzyme discovery and engineering with deep learning using CataPro
Wang, Z., Xie, D., Wu, D., Luo, X., Wang, S., Li, Y., Yang, Y., Li, W., & Zheng, L. (2025) · Nature Communications · doi:10.1038/s41467-025-58038-4 · Cited by 123
PeptideBERT: A Language Model Based on Transformers for Peptide Property Prediction
Guntuboina, C., Das, A., Mollaei, P., Kim, S. W., & Farimani, A. B. (2023) · The Journal of Physical Chemistry Letters · doi:10.1021/acs.jpclett.3c02398 · Cited by 131
A survey of generative AI for de novo drug design: new frontiers in molecule and protein generation
Tang, X., Dai, H., Knight, E., Wu, F., Li, Y., Li, T., & Gerstein, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae338 · Cited by 150
“ProteinVAE [118] utilizes ProtBERT [119] to reduce raw input sequences into latent representations, employing an encoder–decoder framework with position-wise multi-head self-attention to capture long-range dependencies in these sequences.”
Foundation models in bioinformatics
Guo, F., Guan, R., Li, Y., Liu, Q., Wang, X., Yang, C., & Wang, J. (2025) · National Science Review · doi:10.1093/nsr/nwaf028 · Cited by 127
CatPred: a comprehensive framework for deep learning in vitro enzyme kinetic parameters
Boorla, V. S., & Maranas, C. D. (2025) · Nature Communications · doi:10.1038/s41467-025-57215-9 · Cited by 122
Contrastive learning on protein embeddings enlightens midnight zone
(2022) · doi:10.1093/nargab/lqac043 · Cited by 105
“As recommended 89 in the original publication (9), for ProtT5, we only used the encoder part 90 of ProtT5 in half-precision to embed protein sequences.”
BackgroundMethodResultInfluentialTMbed: transmembrane proteins predicted through language model embeddings
Bernhofer, M., & Rost, B. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04873-x · Cited by 109
“We generated embeddings with protein Language Models (pLMs) for our data sets using a transformer-based pLM ProtT5-XL-U50 (short: ProtT5) [34].”
BackgroundMethodInfluentialGhafarollahi, A., & Buehler, M. J. (2024) · Digital Discovery · doi:10.1039/d4dd00013g · Cited by 167
AlphaFold2 reveals commonalities and novelties in protein structure space for 21 model organisms
Bordin, N., Sillitoe, I., Nallapareddy, V., Rauer, C., Lam, S. D., Waman, V. P., Sen, N., Heinzinger, M., Littmann, M., Kim, S., et al. (2023) · Communications Biology · doi:10.1038/s42003-023-04488-9 · Cited by 103
“To determine whether these domains were distant homologues of CATH superfamilies, we used a recently established protein Language Model (pLM, Prot-T5 (24)) to generate protein sequence embeddings of the domains.”
MethodInfluentialSelf-citationAI-Driven Antimicrobial Peptide Discovery: Mining and Generation
Szymczak, P., Zarzecki, W., Wang, J., Duan, Y., Wang, J., Coelho, L. P., de la Fuente‐Núñez, C., & Szczurek, E. (2025) · Accounts of Chemical Research · doi:10.1021/acs.accounts.0c00594 · Cited by 113
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112
BioSeq-Diabolo: Biological sequence similarity analysis using Diabolo
Li, H., & Liu, B. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011214 · Cited by 103
A Transformer-Based Ensemble Framework for the Prediction of Protein–Protein Interaction Sites
Mou, M., Pan, Z., Zhou, Z., Zheng, L., Zhang, H., Shi, S., Li, F., Sun, X., & Zhu, F. (2023) · Research · doi:10.34133/research.0240 · Cited by 106
Fang, Y., Jiang, Y. F., Wei, L., Ma, Q., Ren, Z., Yuan, Q., & Wei, D. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad718 · Cited by 103
PepNN: a deep attention model for the identification of peptide binding sites
Abdin, O., Nim, S., Wen, H., & Kim, P. M. (2022) · Communications Biology · doi:10.1038/s42003-022-03445-2 · Cited by 104
“In addition, the idea of pre-training contextualized language models has recently been adapted to protein biology for the purpose of generating meaningful representations of protein sequences 23,24 .”
MethodInfluentialDeepSTABp: A Deep Learning Approach for the Prediction of Thermal Protein Stability
Jung, F., Frey, K. L., Zimmer, D., & Mühlhaus, T. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24087444 · Cited by 98
“It has been shown that the size of the training dataset correlates with predictive power of the language model effectively learning structure-relevant features that would be not possible to extract from the TPP dataset alone according to size limitations [29].”
BackgroundWang, L., Patena, W., Baalen, K. A. V., Xie, Y., Singer, E. R., Gavrilenko, S., Warren-Williams, M., Han, L., Harrigan, H. R., Hartz, L. D., et al. (2023) · Cell · doi:10.1016/j.cell.2023.06.008 · Cited by 102
PLMSearch: Protein language model powers accurate and fast sequence search for remote homology
Liu, W., Wang, Z., You, R., Xie, C., Wei, H., Xiong, Y., Yang, J., & Zhu, S. (2024) · Nature Communications · doi:10.1038/s41467-024-46808-5 · Cited by 96
SignalP: The Evolution of a Web Server
Nielsen, H., Teufel, F., Brunak, S., & von HEIJNE, G. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4007-4_17 · Cited by 107
AlQuraishi, M., & Sorger, P. K. (2021) · Nature Methods · doi:10.1038/s41592-021-01283-4 · Cited by 96
Genomic language model predicts protein co-regulation and function
Hwang, Y., Cornman, A. L., Kellogg, E. H., Ovchinnikov, S., & Girguis, P. R. (2024) · Nature Communications · doi:10.1038/s41467-024-46947-9 · Cited by 99
Yuan, Q., Xie, J., Xie, J., Zhao, H., & Yang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad117 · Cited by 93
Deep learning for the PSIPRED Protein Analysis Workbench
Buchan, D. W. A., Moffat, L., Lau, A. M., Kandathil, S. M., & Jones, D. T. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae328 · Cited by 106
Graph Neural Networks in Modern AI-Aided Drug Discovery
Zhang, O., Lin, H., Zhang, X., Wang, X., Wu, Z., Ye, Q., Zhao, W., Wang, J., Ying, K., Kang, Y., et al. (2025) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00461 · Cited by 110
Large language models improve annotation of prokaryotic viral proteins
Flamholz, Z. N., Biller, S. J., & Kelly, L. (2024) · Nature Microbiology · doi:10.1038/s41564-023-01584-8 · Cited by 92
“The Transformer_BFD PLM from the ProtTrans 28 project was used via the DeepChainBio/BioTransformers Python package (https://github. com/DeepChainBio/bio-transformers).”
Rules and mechanisms governing G protein coupling selectivity of GPCRs
Masuho, I., Kise, R., Gaínza, P., Moo, E. V., Li, X., Tany, R., Wakasugi-Masuho, H., Correia, B. E., & Martemyanov, K. A. (2023) · Cell Reports · doi:10.1016/j.celrep.2023.113173 · Cited by 92
“The pretrained protein BERT model from the ProtTrans suite 68 was installed using the Transformers library (https://huggingface.co/docs/transformers/index).”
Ni, B., Kaplan, D. L., & Buehler, M. J. (2023) · Chem · doi:10.1016/j.chempr.2023.03.020 · Cited by 92
Explainable Deep Hypergraph Learning Modeling the Peptide Secondary Structure Prediction
Jiang, Y., Wang, R., Feng, J., Jin, J., Liang, S., Li, Z., Yu, Y., Ma, A., Su, R., Zou, Q., et al. (2023) · Advanced Science · doi:10.1002/advs.202206151 · Cited by 92
Liu, S., Rodriguez, J. S., Munteanu, V., Ronkowski, C., Sharma, N. K., Alser, M., Andreace, F., Blekhman, R., Błaszczyk, D., Chikhi, R., et al. (2025) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00376-6 · Cited by 99
Metadata
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Jiangnan University (CN)
- Google (United States) (US)
- Nvidia (United States) (US)
- NVIDIA, Santa Clara, CA, USA
- Seoul National University (KR)
- Oak Ridge National Laboratory (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 7112-7127
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3177500196
- Cited by
- 2,329
- References
- 101
- Citations by year
- 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
- New Faculty Startup Fund and the Creative-Pioneering Researchers Program
- Seoul National University
- Oak Ridge National Laboratory
- U.S. Department of Energy (DE-AC05-00OR22725)
- TensorFlow Research Cloud
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
- MeSH terms
- Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
