The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Open accessPublished version of a preprint
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Fine-tuning has not been extensively explored in previous work5, possibly due to its demanding computational requirements.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“Such models were then applied to protein sequence: the BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model [58] for protein sequence prediction.”
From Computational and artificial intelligence-based methods for antibody development (2023) · cites this paper as [58]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Flagged first, then by citations
2 of the 101 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
- limited, 2019, Press release announcing Supercomputer Fugaku
- rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
- armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
- 10.1110/ps.03128904 · DOI
Cited by
Sorted by citations
Items 201–300 of 1,007
Shin, I., Kang, K., Kim, J., Sel, S., Choi, J., Lee, J.-W., Kang, H.-Y., & Song, G. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05577-6 · Cited by 50
“This study utilized self-supervised learning with two pretraining tasks in AptaTrans: masked token prediction (MTP) and secondary structure prediction (SSP) [23, 24].”
MethodProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention
Li, M., Tan, Y., Ma, X., Zhong, B., Yu, H., Zhou, Z., Ouyang, W., Zhong, B., Hong, L., & Tan, Y. (2024) · neural information processing systems · doi:10.52202/079017-1126 · Cited by 52
Recent Advances in Machine Learning Variant Effect Prediction Tools for Protein Engineering
Horne, J., & Shukla, D. (2022) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.1c04943 · Cited by 48
Deep structured learning for variant prioritization in Mendelian diseases
(2023) · doi:10.1038/s41467-023-39306-7 · Cited by 43
“Architecture 2 (Supplementary Fig.2)alsousesthealteredprotein sequence, evolutionary conservation data, and structured data, but uses a pre-trained “ protein language ” model called ProtT5-XL BFD as a second type of feature extractor forthealtered proteinsequence 32 .”
Nallapareddy, V., Bordin, N., Sillitoe, I., Heinzinger, M., Littmann, M., Waman, V. P., Sen, N., Rost, B., & Orengo, C. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad029 · Cited by 47
Fenoy, E., Edera, A. A., & Stegmayer, G. S. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac232 · Cited by 47
Manfredi, M., Savojardo, C., Martelli, P. L., & Casadio, R. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac678 · Cited by 48
Soylu, N. N., & Sefer, E. (2024) · Current Bioinformatics · doi:10.2174/0115748936283134240109054157 · Cited by 46
Zhu, Y., Liu, Z., Liu, Y., Ji, Z., & Yu, D. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae040 · Cited by 49
Multi-modal deep learning enables efficient and accurate annotation of enzymatic active sites
Wang, X., Yin, X., Jiang, D., Zhao, H., Wu, Z., Zhang, O., Wang, J., Li, Y., Deng, Y., Liu, H., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51511-6 · Cited by 47
ProLLaMA: A Protein Large Language Model for Multitask Protein Language Processing
Lv, L., Lin, Z., Li, H., Liu, Y., Cui, J., Chen, C. Y., Yuan, L., & Tian, Y. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3564914 · Cited by 93
CellSAM: a foundation model for cell segmentation
Marks, M., Israel, U., Dilip, R., Li, Q., Yu, C., Laubscher, E., Iqbal, A., Pradhan, E., Ates, A., Abt, M., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02879-w · Cited by 152
Rapid and automated design of two-component protein nanomaterials using ProteinMPNN
de Haas, R. J., Brunette, N., Goodson, A., Dauparas, J., Yi, S. Y., Yang, E. C., Dowling, Q., Nguyen, H., Kang, A., Bera, A. K., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314646121 · Cited by 42
Prevalent bee venom genes evolved before the aculeate stinger and eusociality
Koludarov, I., Velasque, M., Senoner, T., Timm, T., Greve, C., Hamadou, A. B., Gupta, D. K., Lochnit, G., Heinzinger, M., Vilcinskas, A., et al. (2023) · BMC Biology · doi:10.1186/s12915-023-01656-5 · Cited by 44
“These advances have been transferred to protein sequences by learning to predict masked or missing amino acids using large databases of raw protein sequences as input [122, 123].”
BackgroundMethodInfluentialPang, Y.-H., & Liu, B. (2024) · BMC Biology · doi:10.1186/s12915-023-01803-y · Cited by 38
“Studies [55, 64, 65] have demonstrated that applying protein semantic information extracted from PLMs can facilitate the performance improvement of various prediction tasks.”
BackgroundMethodOptimizing enzyme thermostability by combining multiple mutations using protein language model
Bian, J., Tan, P., Nie, T., Hong, L. Y., & Yang, G. (2024) · mLife · doi:10.1002/mlf2.12151 · Cited by 43
Drug-target interaction/affinity prediction: Deep learning models and advances review
Vefghi, A., Rahmati, Z., & Akbari, M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110438 · Cited by 47
The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modelingPreprint
Cornman, A., West-Roberts, J. A., Camargo, A. P., Roux, S., Beracochea, M., Mirdita, M., Ovchinnikov, S., & Hwang, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.14.607850 · Cited by 52
Geffen, Y., Ofran, Y., & Unger, R. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac474 · Cited by 40
Protein structure-informed bacteriophage genome annotation with Phold
Bouras, G., Grigson, S. R., Mirdita, M., Heinzinger, M., Papudeshi, B., Mallawaarachchi, V., Green, R., Kim, R. S., Mihalia, V., Psaltis, A. J., et al. (2026) · Nucleic Acids Research · doi:10.1093/nar/gkaf1448 · Cited by 71
Jiang, Y., Jiang, L., Akhil, C. S., Wang, D., Zhang, Z., Zhang, W., & Xu, D. H. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad374 · Cited by 42
Modeling protein–ligand interactions for drug discovery in the era of deep learning
Wang, Y. X., Li, Y., Chen, J., & Lai, L. (2025) · Chemical Society Reviews · doi:10.1039/d5cs00415b · Cited by 53
Jiang, T. T., Li, F., & Wang, K. (2023) · The Innovation · doi:10.1016/j.xinn.2023.100487 · Cited by 47
Nanobodies: From Discovery to AI-Driven Design
Zhu, H., & Ding, Y. (2025) · Biology · doi:10.3390/biology14050547 · Cited by 44
Nearest neighbor search on embeddings rapidly identifies distant protein relations
Schütze, K., Heinzinger, M., Steinegger, M., & Rost, B. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.1033775 · Cited by 41
Recent Progress of Protein Tertiary Structure Prediction
Wuyun, Q., Chen, Y., Shen, Y., Cao, Y., Hu, G., Cui, W., Gao, J., & Zheng, W. H. (2024) · Molecules · doi:10.3390/molecules29040832 · Cited by 48
“The inclusion of these deep learning restraints significantly enhanced modeling accuracy, particularly for targets lacking easily identifiable template structures [23].”
BackgroundDepoScope: Accurate phage depolymerase annotation and domain delineation using large language models
Concha-Eloko, R., Stock, M., Bernard De Baets, C., Briers, Y., Sanjuán, R., Domingo‐Calap, P., & Boeckaerts, D. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011831 · Cited by 41
Deciphering 3'UTR Mediated Gene Regulation Using Interpretable Deep Representation Learning
Yang, Y., Li, G. L., Pang, K., Cao, W., Zhang, Z., & Li, X. (2024) · Advanced Science · doi:10.1002/advs.202407013 · Cited by 42
Small data, big challenges: Machine- and deep-learning strategies for data-limited drug discovery
Pallikkavaliyaveetil, N., & Chandrasekaran, S. (2025) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2025.115762 · Cited by 57
Deep neural language modeling enables functional protein generation across familiesPreprint
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.07.18.452833 · Cited by 47
“…proteins are useful for protein informatics tasks such as stability prediction, remote homology detection, and secondary structure prediction 10,24–26 , we show that the latest advances in deep learning-based language modeling can be adopted to generate artificial protein sequences, from scratch,…”
BackgroundNi, B., Kaplan, D. L., & Buehler, M. J. (2024) · Science Advances · doi:10.1126/sciadv.adl4000 · Cited by 51
Improving antibody language models with native pairing
Burbach, S. M., & Briney, B. S. (2024) · Patterns · doi:10.1016/j.patter.2024.100967 · Cited by 40
“36 The vocabulary contained 25 tokens: 1 for each of the 20 amino acids and 5 special tokens: <s>, </s>, <pad>, <unk>, and <mask>.”
GPSFun: geometry-aware protein sequence function predictions with language models
Yuan, Q., Tian, C., Song, Y., Ou, P., Zhu, M., Zhao, H., & Yang, Y. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae381 · Cited by 41
Nielsen, H. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_10 · Cited by 48
Nguyen, T. M., Nguyen, T., & Tran, T. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac269 · Cited by 43
“ProtTrans [14] uses auto-regressive models (Transformer-XL, XLNet) and auto-encoder models (BERT, Albert, Electra, T5) to learn the protein embedding from 2.”
MethodWang, S., Chen, W., Han, P., Li, X., & Song, T. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c01092 · Cited by 39
Pakhrin, S. C., Pokharel, S., Aoki-Kinoshita, K. F., Beck, M. R., Dam, T. K., Caragea, D., & KC, D. B. (2023) · Glycobiology · doi:10.1093/glycob/cwad033 · Cited by 39
CapsNet-MHC predicts peptide-MHC class I binding based on capsule neural networks
Kalemati, M., Darvishi, S., & Koohi, S. (2023) · Communications Biology · doi:10.1038/s42003-023-04867-2 · Cited by 39
Chitkeshwar, A. K. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10117-3 · Cited by 48
Jiang, R., Yue, Z., Shang, L., Wang, D., & Wei, N. (2024) · Metabolic Engineering Communications · doi:10.1016/j.mec.2024.e00248 · Cited by 42
CoCoNat: a novel method based on deep learning for coiled-coil prediction
Madeo, G., Savojardo, C., Manfredi, M., Martelli, P. L., & Casadio, R. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad495 · Cited by 38
Aparicio, J. A. B., Olivares‐Gil, A., Rodríguez, J. J., García‐Osorio, C., & Díez-Pastor, J.-F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102035 · Cited by 41
Predicting enzymatic function of protein sequences with attention
Buton, N., Coste, F., & Cunff, Y. L. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad620 · Cited by 42
Protein Engineering with Lightweight Graph Denoising Neural Networks
Zhong, B., Zheng, L., Wu, B., Tan, Y., Lv, O., Yi, K., Fan, G., & Hong, L. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00036 · Cited by 39
Protein representations: Encoding biological information for machine learning in biocatalysis
Harding-Larsen, D., Funk, J., Madsen, N. G., Gharabli, H., Acevedo-Rocha, C. G., Mazurenko, S., & Welner, D. H. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108459 · Cited by 38
Engineering highly active nuclease enzymes with machine learning and high-throughput screening
Thomas, N., Belanger, D., Xu, C. A., Lee, H., Hirano, K., Iwai, K., Polic, V., Nyberg, K. D., Hoff, K. G., Frenz, L., et al. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101236 · Cited by 41
Khare, E., González‐Obeso, C., Kaplan, D. L., & Buehler, M. J. (2022) · ACS Biomaterials Science & Engineering · doi:10.1021/acsbiomaterials.2c00737 · Cited by 37
Madan, S., Demina, V., Stapf, M., Ernst, O., & Fröhlich, H. (2022) · Patterns · doi:10.1016/j.patter.2022.100551 · Cited by 38
“ProtBERT: Pre-trained Embeddings of Protein Sequences ProtBERT (Elnaggar et al. 2021) is a pre-trained model trained on around two billion protein sequences using a masked language modeling objective (Devlin et al. 2018).”
BackgroundMethodInfluentialLi, G., Yao, S., & Fan, L. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01697 · Cited by 37
AI for organic and polymer synthesis
Hong, X., Yang, Q., Liao, K., Pei, J., Chen, M., Mo, F., Lu, H., Zhang, W.-B., Zhou, H., Chen, J., et al. (2024) · Science China Chemistry · doi:10.1007/s11426-024-2072-4 · Cited by 41
ABodyBuilder3: improved and scalable antibody structure predictions
Kenlay, H., Dreyer, F. A., Cutting, D., Nissley, D. A., & Deane, C. M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae576 · Cited by 44
BioMedGPT: An Open Multimodal Large Language Model for BioMedicine
Luo, Y., Zhang, J., Fan, S., Yang, K., Hong, M., Wu, Y., Qiao, M., & Nie, Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3505955 · Cited by 195
Deep Learning for Antimicrobial Peptides: Computational Models and Databases
Zhou, X., Liu, G., Cao, S., & Lv, J. Z. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00006 · Cited by 37
AI-guided discovery and optimization of antimicrobial peptides through species-aware language model
Bae, D., Kim, M., Seo, J., & Nam, H. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf343 · Cited by 46
Generative latent diffusion language modeling yields anti-infective synthetic peptides
Torres, M. D. T., Chen, T., Wan, F., Chatterjee, P., & de la Fuente‐Núñez, C. (2025) · Cell Biomaterials · doi:10.1016/j.celbio.2025.100183 · Cited by 43
PLM-interact: extending protein language models to predict protein-protein interactions
Liu, D., Young, F., Lamb, K. D., Quiros, A. C., Pancheva, A., Miller, C. J., Macdonald, C., Robertson, D. L., & Yuan, K. (2025) · Nature Communications · doi:10.1038/s41467-025-64512-w · Cited by 65
Saleh, A., Noguchi, Y., Aramayo, R. J., Ivanova, M. E., Stevens, K. M., Montoya, A., Sunidhi, S., Carranza, N. L., Skwark, M. J., & Speck, C. (2022) · Nature Communications · doi:10.1038/s41467-022-30576-1 · Cited by 41
Pakhrin, S. C., Pokharel, S., Pratyush, P., Chaudhari, M., Ismail, H. D., & KC, D. B. (2023) · Journal of Proteome Research · doi:10.1021/acs.jproteome.2c00667 · Cited by 37
Domain loss enabled evolution of novel functions in the snake three-finger toxin gene superfamily
Koludarov, I., Senoner, T., Jackson, T., Dashevsky, D., Heinzinger, M., Aird, S. D., & Rost, B. (2023) · Nature Communications · doi:10.1038/s41467-023-40550-0 · Cited by 38
“In this study, we combined reliable 3D structure predictions from AlphaFold2 14 and embeddings from the pLM ProtT5 18 with traditional bioinformatic analyses and extensive manual microsyntenic analyses.”
InfluentialGong, J., Jiang, L., Chen, Y., Zhang, Y., Li, X., Ma, Z., Fu, Z., He, F., Sun, P., Ren, Z., et al. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad646 · Cited by 37
Chandra, A. A., Sharma, A. K., Dehzangi, A., Tsunoda, T., & Sattar, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-47624-5 · Cited by 38
Djeddi, W. E., Hermi, K., Yahia, S. B., & Diallo, G. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05593-6 · Cited by 37
Grigorakis, K., Ferousi, C., & Topakas, E. (2025) · Catalysts · doi:10.3390/catal15020147 · Cited by 39
Machine learning in predictive biocatalysis: A comparative review of methods and applications
Tripathi, N., Hérisson, J., & Faulon, J. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108698 · Cited by 37
CF-DTI: coarse-to-fine feature extraction for enhanced drug-target interaction prediction
Qian, Y., Wang, Q., Yin, L., & Lu, A. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00370-6 · Cited by 36
Yeung, W., Zhou, Z., Li, S., & Kannan, N. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbac599 · Cited by 35
“Each data point represents a distinct protein language model and the color indicates models from the same family where six ESM2 [10] models are shown in green, four ProtTrans [11] models in red, one ESM1b [9] model in gold and three ESM1 [9] models in blue.”
InfluentialRai, B. K., Apgar, J. R., & Bennett, E. M. (2023) · Scientific Reports · doi:10.1038/s41598-023-28841-4 · Cited by 34
“Recent publications 33,34 utilizing embeddings generated from protein language models 34–36 , trained on large corpus of protein sequence data, have shown that a high-level of generalization can be achieved in a limited training data regime.”
BackgroundZhang, X. J., Hu, X., Zhang, T., Yang, L., Liu, C., Xu, N., Wang, H., & Sun, W. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae404 · Cited by 38
Generative AI: driving productivity and scientific breakthroughs in pharmaceutical R&D
Doron, G., Genway, S., Roberts, M., & Jasti, S. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104272 · Cited by 39
RNA-protein interaction prediction using network-guided deep learning
Liu, H., Jian, Y., Zeng, C., & Zhao, Y. (2025) · Communications Biology · doi:10.1038/s42003-025-07694-9 · Cited by 35
Democratizing protein language model training, sharing and collaboration
Su, J., Li, Z., Tao, T., Han, C., He, Y., Dai, F., Yuan, Q., Gao, Y., Si, T., Zhang, X., et al. (2026) · Nature Biotechnology · doi:10.1038/s41587-025-02859-7 · Cited by 35
Pei, H., Li, J., Ma, S., Jiang, J., Li, M., Zou, Q., & Lv, Z. (2023) · Applied Sciences · doi:10.3390/app13052858 · Cited by 38
Wang, D., Pourmirzaei, M., Abbas, U. L., Zeng, S., Manshour, N., Esmaili, F., Poudel, B., Jiang, Y., Shao, Q., Chen, J., et al. (2025) · Advanced Science · doi:10.1002/advs.202404212 · Cited by 46
Phosformer: an explainable transformer model for protein kinase-specific phosphorylation predictions
Zhou, Z., Yeung, W., Gravel, N., Salcedo, M. V., Soleymani, S., Li, S., & Kannan, N. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad046 · Cited by 34
The promises of large language models for protein design and modeling
Valentini, G., Malchiodi, D., Gliozzo, J., Mesiti, M., Soto-Gomez, M., Cabri, A., Reese, J., Casiraghi, E., & Robinson, P. N. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1304099 · Cited by 40
Mullin, M. J., McClory, J., Haynes, W. A., Grace, J., Robertson, N. J., & Heeke, G. V. (2024) · mAbs · doi:10.1080/19420862.2024.2341443 · Cited by 36
Accurate prediction of antibody function and structure using bio-inspired antibody language model
Jing, H., Gao, Z., Xu, S. Y., Shen, T., Peng, Z., He, S., You, T., Ye, S., Lin, W. J., & Sun, S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae245 · Cited by 35
“Recent evidence suggests that language models possess a potent capability for capturing semantic information encoded within input amino acid sequences, thereby facilitating the prediction of structure and function [8, 13, 26].”
AutoPeptideML: a study on how to build more trustworthy peptide bioactivity predictors
Fernández-Díaz, R., Cossio-Pérez, R., Agoni, C., Lam, H. T., López, V., & Shields, D. C. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae555 · Cited by 35
Machine Learning Approaches to TCR Repertoire Analysis
Katayama, Y., Yokota, R., Akiyama, T., & Kobayashi, T. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.858057 · Cited by 35
Hu, F., Hu, Y., Zhang, W., Huang, H., Pan, Y. F., & Yin, P. (2023) · Advanced Science · doi:10.1002/advs.202301223 · Cited by 33
Superior protein thermophilicity prediction with protein language model embeddings
Haselbeck, F., John, M., Zhang, Y., Pirnay, J., Werner, J., Costa, R. D., & Grimm, D. G. (2023) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqad087 · Cited by 35
“These hybrid latent representations are further processed by an LSTM and Bi-LSTM, respectively.”
BackgroundInfluentialSong, Y., Yuan, Q., Zhao, H., & Yang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad360 · Cited by 34
Recent Advances and Challenges in Protein Structure Prediction
Peng, C., Liang, F., Xia, Y., Zhao, K., Hou, M., & Zhang, G. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01324 · Cited by 42
Kim, D. N., McNaughton, A. D., & Kumar, N. (2024) · Bioengineering · doi:10.3390/bioengineering11020185 · Cited by 35
“Other Other LM ( ProtTrans [110])-based prediction methods of the binding affinity of Abs (BCR) and Ags include DeepAIR [111].”
MethodFusPB-ESM2: Fusion model of ProtBERT and ESM-2 for cell-penetrating peptide prediction
Zhang, F., Li, J., Wen, Z., & Fang, C. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108098 · Cited by 35
Basith, S., Pham, N. T., Manavalan, B., & Lee, G. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.133085 · Cited by 33
Surface-based multimodal protein–ligand binding affinity prediction
Xu, S., Shen, L., Zhang, M., Jiang, C., Zhang, X., Xu, Y., Liu, J., & Liu, X. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae413 · Cited by 35
Wyler, E., Lauber, C., Manukyan, A., Deter, A., Quedenau, C., Alves, L. G. T., Wylezich, C., Borodina, T. A., Seitz, S., Altmüller, J., et al. (2024) · Environment International · doi:10.1016/j.envint.2024.108875 · Cited by 32
Qian, Y., Su, L.-J., Xu, M., & Tang, L. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3523772 · Cited by 33
Massive experimental quantification allows interpretable deep learning of protein aggregation
Thompson, M., Martín, M., Olmo, T. S., Rajesh, C., Koo, P. K., Bolognesi, B., & Lehner, B. (2025) · Science Advances · doi:10.1126/sciadv.adt5111 · Cited by 37
Gujral, O. S., Bafna, M., Alm, E. J., & Berger, B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2506316122 · Cited by 49
Parameter-efficient fine-tuning on large protein language models improves signal peptide predictionPreprint
Zeng, S., Wang, D., Jiang, L., & Xu, D. H. (2024) · Genome Research · doi:10.1101/gr.279132.124 · Cited by 34
“In contrast to its predecessors, SignalP 6.0 (Teufel et al. 2022) stands out as a remarkable tool capable of predicting all five types of SPs (Sec/SPI, Sec/SPII, Tat/SPI, Tat/SPII, Sec/SPIII) throughProtTrans (Elnaggaret al. 2022), arobustproteinlanguagemodel(PLM)pretrainedontheUniRef100 data set…”
InfluentialEPIC-TRACE: predicting TCR binding to unseen epitopes using attention and contextualized embeddings
Korpela, D., Jokinen, E., Dumitrescu, A., Huuhtanen, J., Mustjoki, S. M., & Lähdesmäki, H. J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad743 · Cited by 32
“If the full TCR cannot be constructed, only the CDR3 region is embedded with ProtBERT and no further extraction is done.”
MethodInfluentialMultimodal pretraining for unsupervised protein representation learning
Nguyen, V. T. D., & Hy, T. S. (2024) · Biology Methods and Protocols · doi:10.1093/biomethods/bpae043 · Cited by 33
Pre-training with fractional denoising to enhance molecular property prediction
Ni, Y., Feng, S., Hong, X., Sun, Y., Ma, W., Ma, Z.-M., Ye, Q., & Lan, Y. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00900-z · Cited by 37
Buehler, M. J. (2022) · Accounts of Chemical Research · doi:10.1021/acs.accounts.2c00330 · Cited by 32
Is Protein BLAST a thing of the past?
Al-Fatlawi, A. H., Menzel, M., & Schroeder, M. (2023) · Nature Communications · doi:10.1038/s41467-023-44082-5 · Cited by 32
Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.31.526427 · Cited by 37
Expert-guided protein language models enable accurate and blazingly fast fitness prediction
Marquet, C., Schlensok, J., Abakarova, M., Rost, B., & Laine, E. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae621 · Cited by 33
Metadata
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Jiangnan University (CN)
- Google (United States) (US)
- Nvidia (United States) (US)
- NVIDIA, Santa Clara, CA, USA
- Seoul National University (KR)
- Oak Ridge National Laboratory (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 7112-7127
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3177500196
- Cited by
- 2,329
- References
- 101
- Citations by year
- 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
- New Faculty Startup Fund and the Creative-Pioneering Researchers Program
- Seoul National University
- Oak Ridge National Laboratory
- U.S. Department of Energy (DE-AC05-00OR22725)
- TensorFlow Research Cloud
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
- MeSH terms
- Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
