The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Open accessPublished version of a preprint
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Fine-tuning has not been extensively explored in previous work5, possibly due to its demanding computational requirements.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“Such models were then applied to protein sequence: the BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model [58] for protein sequence prediction.”
From Computational and artificial intelligence-based methods for antibody development (2023) · cites this paper as [58]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Research works printed but not linked: 5
- john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
- limited, 2019, Press release announcing Supercomputer Fugaku
- rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
- armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
- 10.1110/ps.03128904 · DOI
Cited by
Sorted by citations
Items 301–400 of 1,007
Pratyush, P., Pokharel, S., Saigo, H., & KC, D. B. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05164-9 · Cited by 29
“The advances in Natural Language Processing (NLP) gained by the development of newer language models have been transferred to protein sequences by learning to predict masked or missing amino acids using a large corpus of protein sequences [21–23].”
BackgroundMethodInfluentialDomain-PFP allows protein function prediction using function-aware domain embedding representations
Ibtehaz, N., Kagaya, Y., & Kihara, D. (2023) · Communications Biology · doi:10.1038/s42003-023-05476-9 · Cited by 31
“These PLMs were trained on unsupervised tasks such as predicting a segment of masked residues given the rest of the protein 30 or predicting the next residue from all the residues before it 50 , on a large protein sequence dataset, e.g. the entire UniProt.”
Shen, J., Yu, Q., Chen, S., Tan, Q., Li, J., & Li, Y. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00576-2 · Cited by 32
Advancing bioinformatics with language models: components, applications, and perspectives
Liu, J., Yang, M., Yu, Y., Xu, H., Wang, T., Li, K., & Zhou, X. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag367 · Cited by 66
MDF-DTA: A Multi-Dimensional Fusion Approach for Drug-Target Binding Affinity Prediction
Ranjan, A., Bess, A., Alvin, C., & Mukhopadhyay, S. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00310 · Cited by 32
Chen, N., Yu, J., Liu, Z., Wang, F., Li, X., & Wong, K. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae308 · Cited by 32
“As a result, the embeddings matrix for each sequence is X lm ∈ R L max × 1024 , where lm = sv. WealsoutilizethePortTransmodel,aseriesofproteinlanguage models pre-trained based on BERT, Albert, Electra and T5 [37].”
InfluentialTemBERTure: advancing protein thermostability prediction with deep learning and attention mechanisms
Rodella, C., Lazaridi, S., & Lemmin, T. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae103 · Cited by 31
Cai, T., Lim, H., Abbu, K. A., Qiu, Y., Nussinov, R., & Xie, L. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01285 · Cited by 31
Xie, S., Xie, X., Zhao, X. Y., Liu, F., Wang, Y., Ping, J., & Ji, Z. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad261 · Cited by 32
BEND: Benchmarking DNA Language Models on biologically meaningful tasksPreprint
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.12570 · Cited by 54
Multi-task bioassay pre-training for protein-ligand binding affinity prediction
Yan, J., Ye, Z., Yang, Z., Lu, C., Zhang, S., Liu, Q., & Qiu, J. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad451 · Cited by 32
“For small molecules and proteins, a series of selfsupervised pre-training methods based on molecular graphs [14; 15; 16; 17] and protein sequences [18; 19; 20; 21] have been proposed, respectively.”
MethodArif, M., Musleh, S., Fida, H., & Alam, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-67433-8 · Cited by 30
Hao, Y., Liu, X., Fu, H., Shao, X., & Cai, W. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae497 · Cited by 32
“Given the superior performance of CNN in ProtTrans downstream tasks (Elnaggar et al. 2021, Du et al. 2023), we used ProtT5-CNN to compare its performance with ProtT5-GAT, aiming to observe the significance of structural information.”
BackgroundMethodInfluentialPathak, A., Theagarajan, R., Rizqi, M. M., Nugraha, A. S., Boruah, T., Pratiksha, Kumar, H., Naik, B., Yadav, S., Jha, A. K., et al. (2025) · Discover Molecules · doi:10.1007/s44345-025-00037-5 · Cited by 34
Wang, X. R., Ding, Z., Wang, R., & Lin, X. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbac631 · Cited by 33
Protein embedding based alignment
Iovino, B. G., & Ye, Y. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05699-5 · Cited by 29
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41
M-Ionic: prediction of metal-ion-binding sites from sequence using residue embeddings
Shenoy, A., Kalakoti, Y., Sundar, D., & Elofsson, A. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btad782 · Cited by 29
Zhao, F., Qiu, J., Xiang, D., Jiao, P., Cao, Y., Xu, Q., Qiao, D., Xu, H., & Cao, Y. (2024) · PeerJ · doi:10.7717/peerj.17729 · Cited by 31
“In addition to Bi-STLM, we introduced two state-of-the-art protein language models ProtTrans ( Elnaggar et al., 2021 ) and ESM2 ( Lin et al., 2023 ), and compared the performance of these three protein language models with the three hyperparameters identified above.”
Chung, C., Chien, C.-Y., Tang, Y., Wu, L.-C., Hsu, J. B., Lu, J., Lee, T.-Y., Bai, C., & Horng, J. (2024) · iScience · doi:10.1016/j.isci.2024.110718 · Cited by 34
Approaching Optimal pH Enzyme Prediction with Large Language Models
Zaretckii, M., Buslaev, P., Kozlovskii, I., Morozov, A., & Popov, P. A. (2024) · ACS Synthetic Biology · doi:10.1021/acssynbio.4c00465 · Cited by 29
Ontology Embedding: A Survey of Methods, Applications and Resources
Chen, J., Mashkova, O., Zhapa-Camacho, F., Hoehndorf, R., He, Y., & Horrocks, I. R. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3559023 · Cited by 48
Learning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
Pope, Q., Varma, R., Tataru, C. A., David, M. M., & Fern, X. Z. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011353 · Cited by 30
From Code to Life: The AI‐Driven Revolution in Genome Editing
Li, Z., Khan, W. U., Bai, G., Chao, D., Wang, J., Zhang, Y., Wang, C., Zhang, H., Wang, W., Luo, M., et al. (2025) · Advanced Science · doi:10.1002/advs.202417029 · Cited by 31
GOProFormer: A Multi-Modal Transformer Method for Gene Ontology Protein Function Prediction
Kabir, A., & Shehu, A. (2022) · Biomolecules · doi:10.3390/biom12111709 · Cited by 29
Survey of Protein Sequence Embedding Models
Tran, C. T. T., Khadkikar, S., & Porollo, A. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24043775 · Cited by 31
Seq-InSite: sequence supersedes structure for protein interaction site prediction
Hosseini, S., Golding, G. B., & Ilie, L. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btad738 · Cited by 26
Biomaterials in cellular agriculture and plant-based foods for the future
Gordon, E. B., Choi, I., Amanipour, A., Hu, Y., Nikkhah, A., Koysuren, B., Jones, C., Nitin, N., Ovissipour, R., Buehler, M. J., et al. (2025) · Nature Reviews Materials · doi:10.1038/s41578-025-00800-7 · Cited by 33
Hu, Y., & Buehler, M. J. (2022) · ACS Nano · doi:10.1021/acsnano.2c07681 · Cited by 26
Yeung, W., Zhou, Z., Mathew, L., Gravel, N., Taujale, R., O’Boyle, B. M., Salcedo, M. V., Venkat, A., Lanzilotta, W. N., Li, S., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbac619 · Cited by 28
“Adapting these methods toward biological data, protein language models are trained on millions of biologically observed protein sequences in a self-supervised manner, without annotations [1, 2].”
The Application of Artificial Intelligence Accelerates G Protein-Coupled Receptor Ligand Discovery
Chen, W., Song, C., Leng, L., Zhang, S., & Chen, S. (2023) · Engineering · doi:10.1016/j.eng.2023.09.011 · Cited by 28
Ultra-fast protein structure prediction to capture effects of sequence variation in mutation moviesPreprint
Weißenow, K., Heinzinger, M., Steinegger, M., & Rost, B. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.14.516473 · Cited by 29
Shanehsazzadeh, A., Alverio, J., Kasun, G. W., Levine, S., Calman, I., Khan, J. A., Chung, C., Diaz, N., Luton, B. K., Tarter, Y., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.08.570889 · Cited by 27
Chen, T., Vure, P., Pulugurta, R., & Chatterjee, P. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.03.583201 · Cited by 31
Language models for the prediction of SARS-CoV-2 inhibitors
Blanchard, A. E., Gounley, J., Bhowmik, D., Chandra Shekar, M., Lyngaas, I., Gao, S., Yin, J., Tsaris, A., Wang, F., & Glaser, J. (2022) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420221121804 · Cited by 30
Protein embeddings improve phage-host interaction prediction
Gonzales, M. E. M., Ureta, J. C., & Shrestha, A. M. S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289030 · Cited by 28
“In an attempt to address these gaps, our study seeks to contribute the following: • We extensively tested different protein language models to automatically generate dense embeddings of RBP sequences.”
Li, B., & Ming, D. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05820-8 · Cited by 27
Protein embeddings predict binding residues in disordered regions
(2024) · doi:10.1038/s41598-024-64211-4 · Cited by 27
“The decrease in estimates for precision and recall from validation to test set were similar for the random baseline.”
BackgroundMethodResultInfluentialT-cell receptor binding prediction: A machine learning revolution
Weber, A., Pélissier, A., & Martínez, M. R. (2024) · ImmunoInformatics · doi:10.1016/j.immuno.2024.100040 · Cited by 28
“For instance, TCRconv [77] employs ProtBERT [70] embeddings as input and processes them through a convolutional and a linear layer for the downstream task of epitope classification.”
BackgroundMethodWang, J., Liu, Y., & Tian, B. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-024-00920-2 · Cited by 28
Peptide-Aware Chemical Language Model Successfully Predicts Membrane Diffusion of Cyclic Peptides
Feller, A. L., & Wilke, C. O. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01441 · Cited by 33
PROTSOLM: Protein Solubility Prediction with Multi-modal Features
Tan, Y., Zheng, J., Liang, H., & Zhong, B. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822310 · Cited by 27
“The second category contains fine-tuned protein language models, including different versions of ESM2 [23], ProtBert [56], and encoder-decoder architecture models: ProtT5 [56], Ankh [57].”
Cai, J., Yan, J., Un, C.-W., Wang, Y., Campbell‐Valois, F., & Siu, S. W. I. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01749 · Cited by 27
A Robust Drug–Target Interaction Prediction Framework with Capsule Network and Transfer Learning
Huang, Y., Huang, H., Chen, Y.-G., Lin, Y.-C.-D., Yao, L., Lin, T., Leng, J., Chang, Y., Zhang, Y., Zhu, Z., et al. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms241814061 · Cited by 28
“However, accurately predicting the complete 3D structure of a protein solely from its sequence using BERT-based models remains a complex and challenging problem [17,38–41].”
BackgroundMethodInfluentialEnhancing Antibody Language Models with Structural InformationPreprint
Barton, J., Galson, J. D., & Leem, J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.12.569610 · Cited by 38
Sen, N., Anishchenko, I., Bordin, N., Sillitoe, I., Velankar, S., Baker, D., & Orengo, C. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac187 · Cited by 25
“Some of these (41) could be putatively assigned to CATH superfamilies using more advanced sequence search strategies and structure comparisons using the predicted structure (see Supplementary Text 10 available online at http://bib. oxfordjournals.org/) [ 64, 65].”
Enhancing missense variant pathogenicity prediction with protein language models using VariPred
Lin, W., Wells, J., Wang, Z., Orengo, C., & Martin, A. C. R. (2024) · Scientific Reports · doi:10.1038/s41598-024-51489-7 · Cited by 34
“The embeddings of these sequence-based pre-trained models have been shown to encode useful bio-physical information, such as residue conservation 20 amino acid hydrophobicity, protein structure class 16 and protein functional properties 21 .”
MethodResultInfluentialYan, Z., Ge, F., Liu, Y., Zhang, Y., Li, F., Song, J., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02019 · Cited by 25
Contrastive learning of T cell receptor representations
Nagano, Y., Pyo, A. G. T., Milighetti, M., Henderson, J., Shawe-Taylor, J. S., Chain, B., & Tiffeau-Mayer, A. (2025) · Cell Systems · doi:10.1016/j.cels.2024.12.006 · Cited by 34
Artificial intelligence-driven computational methods for antibody design and optimization
Vecchietti, L. F., Wijaya, B. N., Armanuly, A., Hangeldiyev, B., Jung, H., Lee, S., Cha, M., & Kim, H. M. (2025) · mAbs · doi:10.1080/19420862.2025.2528902 · Cited by 32
Ahmad, B., Ouahada, K., & Hamam, H. (2026) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.12.033 · Cited by 32
Learning the language of protein-protein interactions
(2026) · doi:10.1038/s41467-025-67971-3 · Cited by 26
Li, S., Zhou, J., Xu, T., Huang, L., Wang, F., Xiong, H., Huang, W., Dou, D., & Xiong, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3314502 · Cited by 33
Language model-based B cell receptor sequence embeddings can effectively encode receptor specificity
Wang, M., Patsenker, J., Li, H., Kluger, Y., & Kleinstein, S. H. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad1128 · Cited by 28
DeepSS2GO: protein function prediction from secondary structure
Song, F. V., Su, J., Huang, S., Zhang, N., Li, K., Ni, M., & Liao, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae196 · Cited by 26
“Given that this algorithm utilizes the ESM-1b [21] and Prottrans [48] pre-trained models, it is constrained by protein length (less than 1024).”
Yao, L., Xie, P., Guan, J., Chung, C., Zhang, W., Deng, J., Huang, Y., Chiang, Y., & Lee, T.-Y. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae460 · Cited by 27
Umami-gcForest: Construction of a predictive model for umami peptides based on deep forest
Ji, S., Wu, J., An, F., Lou, M., Zhang, T., Guo, J., Wu, P., Zhu, Y. C., & Wu, R. (2024) · Food Chemistry · doi:10.1016/j.foodchem.2024.141826 · Cited by 25
Heinzinger, M., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102986 · Cited by 27
Peng, X., Wang, X., Guo, Y., Ge, Z., Li, F., Gao, X., & Song, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac215 · Cited by 24
“The first stage of RBP-TSTL involved use of the knowledge transferred from a self-supervised pretrained model named ProtT5-XL [43].”
Jiang, Z., Shen, Y.-Y., & Liu, R. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011428 · Cited by 25
Seo, S., Choi, J., Choi, S., Lee, J., Park, C., & Park, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107257 · Cited by 25
Yang, Y., Jerger, A., Feng, S., Wang, Z., Brasfield, C., Cheung, M. S., Zucker, J., & Guan, Q. (2024) · Communications Biology · doi:10.1038/s42003-024-07359-z · Cited by 24
Yue, P., Wu, J., Sun, Y., Zhang, Y., Wang, Q., & Shao, S. (2025) · Nature Communications · doi:10.1038/s41467-025-56526-1 · Cited by 28
Geng, A., Luo, Z., Li, A., Zhang, Z., Zou, Q., Wei, L., & Cui, F. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02072 · Cited by 24
Chen, S., Yan, K., Li, X., & Liu, B. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3540291 · Cited by 24
Lobo, F., González, M. S., Boto, A., & de la Lastra, J. M. P. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms241210270 · Cited by 26
“From this library, we used the embedders: BeplerEmbedder (Bepler), PLUSRNNEmbedder (PlusRNN), SeqVecEmbedder (SeqVec), ESM1bEmbedder (ESM1b), ProtTransBertBFDEmbedder (Prottrans), and ProtTransT5BFDEmbedder (ProtT5).”
MethodInfluentialGhosh, S., & Mitra, P. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107955 · Cited by 24
Prediction of antibiotic resistance mechanisms using a protein language model
Yagimoto, K., Hosoda, S., Sato, M., & Hamada, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae550 · Cited by 25
Genome-wide prediction of disease variants with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2022) · doi:10.1101/2022.08.25.505311 · Cited by 24
Erdős, G., & Dosztányi, Z. (2024) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2024.102950 · Cited by 28
Zhang, C., Wang, Q., Li, Y., Teng, A., Hu, G., Wuyun, Q., & Zheng, W. H. (2024) · Biomolecules · doi:10.3390/biom14121531 · Cited by 29
A unified evolution-driven deep learning framework for virus variation driver prediction
Nie, Z., Liu, X., Chen, J., Wang, Z., Liu, Y., Si, H.-R., Dong, T., Xu, F., Song, G., Wang, Y., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00966-9 · Cited by 25
Guan, J., Xie, P., Meng, D., Yao, L., Yu, D., Chiang, Y., Lee, T.-Y., & Wang, J. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.05.039 · Cited by 26
Li, Y., Xu, X., Zhang, X., Xu, Z., Zhao, J., Zhu, R., Wang, Z., Ran, W., Zhao, W., Yan, N., et al. (2025) · Gut Microbes · doi:10.1080/19490976.2025.2523811 · Cited by 28
Debnath, K., Rana, P., & Ghosh, P. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf491 · Cited by 28
Learning the local landscape of protein structures with convolutional neural networks
(2021) · doi:10.1007/s10867-021-09593-6 · Cited by 23
“In fact, it has been shown that masked residues can be predicted entirely from sequence data [9].”
BackgroundZheng, J., Xiao, X., & Qiu, W. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.859188 · Cited by 25
“This paper motivates our work, that instead of using amino acid physic-chemical properties to encode words and perform clustering, we can vectorization drugs and protein by using advanced methods such as word2vec and ProtBert(Elnaggar et al., 2021), which could map every word (amino acids are regarded as words) into t…”
MethodInfluentialLuu, R. K., Arevalo, S. E., Lu, W., Ni, B., Yang, Z., Shen, S. C.-Y., Berkovich, J., Hsu, Y.-C., Zan, S., & Buehler, M. J. (2024) · An MIT Exploration of Generative AI · doi:10.21428/e4baedd9.33bd7449 · Cited by 26
Sensitive clustering of protein sequences at tree-of-life scale using DIAMOND DeepClustPreprint
Buchfink, B., Ashkenazy, H., Reuter, K., Kennedy, J. A., & Drost, H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.24.525373 · Cited by 23
Training Compute-Optimal Protein Language ModelsPreprint
Cheng, X., Chen, B., Li, P., Gong, J., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.06.597716 · Cited by 28
Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention
Bhattacharya, N., Thomas, N., Rao, R., Dauparas, J., Koo, P. K., Baker, D., Song, Y. S., & Ovchinnikov, S. (2021) · Biocomputing 2022 · doi:10.1142/9789811250477_0004 · Cited by 20
Gonzales, M. E. M., Ureta, J. C., & Shrestha, A. M. S. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btaf016 · Cited by 23
A Review for Artificial Intelligence Based Protein Subcellular Localization
Xiao, H., Zou, Y., Wang, J., & Wan, S. (2024) · Biomolecules · doi:10.3390/biom14040409 · Cited by 24
Evaluating large language models for annotating proteins
Vitale, R., Bugnon, L. A., Fenoy, E., Milone, D. H., & Stegmayer, G. S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae177 · Cited by 22
“ProtTrans [14] (https://github.com/agemagician/ProtTrans): this model was developed using several Transformers.”
MethodpLM4CPPs: Protein Language Model-Based Predictor for Cell Penetrating Peptides
Kumar, N., Du, Z., & Li, Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01338 · Cited by 22
Vongsouthi, V., Georgelin, R., Matthews, D. S., Saunders, J. W., Lee, B. M., Ton, J., Damry, A. M., Frkic, R. L., Spence, M. A., & Jackson, C. J. (2025) · Science Advances · doi:10.1126/sciadv.ads8318 · Cited by 22
Protein functional site annotation using local structure embeddings
Derry, A., Tartici, A., & Altman, R. B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2513219122 · Cited by 21
Protein Structural Phylogenetics
Puente-Lelièvre, C., Malik, A. J., & Douglas, J. (2025) · Genome Biology and Evolution · doi:10.1093/gbe/evaf139 · Cited by 27
Large-scale predicting protein functions through heterogeneous feature fusion
Zheng, R., Huang, Z., & Deng, L. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad243 · Cited by 23
Pokharel, S., Pratyush, P., Ismail, H. D., Ma, J., & KC, D. B. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms242116000 · Cited by 22
“These L × 1 dimensional embeddings are then processed by a feed-forward neural network (FFNN) in the subsequent level (level-1) that classifies the membership status (positive or negative) of the site of interest (S/T).”
BackgroundMethodInfluentialIn silico protein function prediction: the rise of machine learning-based approaches
Chen, J., Gu, Z., Lai, L., & Pei, J. (2023) · Medical Review · doi:10.1515/mr-2023-0038 · Cited by 23
“They tried to capture the biophysical features of protein sequences and veri fi ed the advantages of these embedded features on downstream tasks such as protein secondary structure prediction and protein subcellular localization prediction [59].”
BackgroundCollaPPI: A Collaborative Learning Framework for Predicting Protein-Protein Interactions
Ma, W., Bi, X., Jiang, H., Zhang, S., & Wei, Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3375621 · Cited by 22
Cellular Chaperone Function of Intrinsically Disordered Dehydrin ERD14
Murvai, N., Kalmár, L., Szabó, B., Schád, É., Micsonai, A., Kardos, J., Buday, L., Han, K., Tompa, P., & Tantos, Á. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22126190 · Cited by 23
Mohseni Behbahani, Y., Laine, E., & Carbone, A. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad231 · Cited by 23
“…major breakthrough in natural language processing [14, 71], the concept has been transferred to proteins through protein language models (pLMs) [58, 17, 4, 26]. pLMs learn the fundamental properties and mechanisms of natural protein diversity by reconstructing some masked or the next amino…”
BackgroundDeep embeddings to comprehend and visualize microbiome protein space
Odrzywolek, K., Karwowska, Z., Majta, J., Byrski, A., Milanowska-Zabel, K., & Kosciolek, T. (2022) · Scientific Reports · doi:10.1038/s41598-022-14055-7 · Cited by 22
“Moreover, we have chosen the LSTM architecture as it gave the best results in Remote Homology detection in the TAPE benchmark 39 and achieved superior performance over Transformer-based architecture in the ProtTrans benchmark 44 .”
MethodResultZhou, K., Lei, C., Zheng, J., Huang, Y., & Zhang, Z. (2023) · Plant Methods · doi:10.1186/s13007-023-01119-6 · Cited by 26
Explaining protein–protein interactions with knowledge graph-based semantic similarity
Sousa, R. T., Silva, S., & Pesquita, C. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108076 · Cited by 26
TAWFN: a deep learning framework for protein function prediction
Meng, L., & Wang, X. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae571 · Cited by 22
Leveraging ancestral sequence reconstruction for protein representation learning
Matthews, D. S., Spence, M. A., Mater, A. C., Nichols, J., Pulsford, S. B., Sandhu, M., Kaczmarski, J. A., Miton, C. M., Tokuriki, N., & Jackson, C. J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00935-2 · Cited by 22
Metadata
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Jiangnan University (CN)
- Google (United States) (US)
- Nvidia (United States) (US)
- NVIDIA, Santa Clara, CA, USA
- Seoul National University (KR)
- Oak Ridge National Laboratory (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 7112-7127
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3177500196
- Cited by
- 2,329
- References
- 101
- Citations by year
- 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
- New Faculty Startup Fund and the Creative-Pioneering Researchers Program
- Seoul National University
- Oak Ridge National Laboratory
- U.S. Department of Energy (DE-AC05-00OR22725)
- TensorFlow Research Cloud
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
- MeSH terms
- Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
