Cited by 2,329Machine Learning in BioinformaticsProtein Structure and Dynamicsvaccines and immunoinformatics approaches

The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.

ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Open access

Elnaggar, Ahmed, Heinzinger, Michael, Dallago, Christian, Rehawi, Ghalia, Wang, Yu, Jones, Llion, Gibbs, Tom, Feher, Tamas, Angerer, Christoph, Steinegger, Martin, Bhowmik, Debsindhu, Rost, Burkhard·2022·IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI 10.1109/tpami.2021.3095381

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.

Integrity signals

Registry

Crossref · 10.1109/tpami.2021.3095381

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
The Protein Data Bank
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Untitled work
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213

Flagged first, then by citations

Show all 101 references

2 of the 101 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
limited, 2019, Press release announcing Supercomputer Fugaku
rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
10.1110/ps.03128904 · DOI

Sorted by citations

Items 801–900 of 1,007

SLAM: Structure-aware lysine β-hydroxybutyrylation prediction with protein language model
Qin, Z., Liu, H., Zhao, P., Wang, K., Ren, H., Miao, C., Li, J., Chen, Y., & Chen, Z. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.135741 · Cited by 7
Pair-EGRET: enhancing the prediction of protein–protein interaction sites through graph attention networks and protein language models
Alam, R., Mahbub, S., & Bayzid, M. S. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae588 · Cited by 8
Multi-Modal CLIP-Informed Protein Editing
Yin, M., Zhou, H., Zhu, Y., Lin, M., Wu, Y., Wu, J., Xu, H., Hsieh, C., Hou, T., Chen, J., et al. (2024) · Health Data Science · doi:10.34133/hds.0211 · Cited by 13
SCREEN: A Graph-based Contrastive Learning Tool to Infer Catalytic Residues and Assess Enzyme Mutations
Pan, T., Bi, Y., Wang, X., Zhang, Y., Webb, G. I., Gasser, R. B., Kurgan, L., & Song, J. (2024) · Genomics Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzae094 · Cited by 7
ProG-SOL: Predicting Protein Solubility Using Protein Embeddings and Dual-Graph Convolutional Networks
Li, G., Zhang, N., & Fan, L. (2025) · ACS Omega · doi:10.1021/acsomega.4c09688 · Cited by 7
Adverse Outcome Pathway and Machine Learning to Predict Drug Induced Seizure Liability
Lane, T. R., Snyder, S. H., Harris, J. S., Urbina, F., & Ekins, S. (2025) · ACS Chemical Neuroscience · doi:10.1021/acschemneuro.5c00177 · Cited by 7
Large-scale protein clustering in the age of deep learning
Pereira, J., Pantolini, L., Durairaj, J., & Schwede, T. F. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103078 · Cited by 7
MultiPep-DLCL: recognition of multifunctional therapeutic peptides through deep learning with label-sequence contrastive learning
Li, T., Fan, H., Zhao, J., Yang, X., & Xia, J. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf274 · Cited by 8
“In contrast, BERT and GPT models primarily rely on the encoder or decoder components of the Transformer for unidirectional context learning, but true bidirectionality of contextual embedding is crucial in peptide sequence prediction [54, 55].”
DAM-Faster RCNN: few-shot defect detection method for wood based on dual attention mechanism
Tong, X., Liang, Z., Qin, M., Liu, F., Yang, J., Xiao, H., & Dai, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-05479-y · Cited by 7
DCGCN: Dual-Channel Graph Convolutional Network-Based Drug–Target Interaction Prediction Method with 3D Molecular Structure
Sun, C. Y., Shen, Y., Xu, M. G., Wang, Q., Shang, T., Yin, Q., Wang, Y., & Zhang, B. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01012 · Cited by 7
EnsemPred-ACP: Combining machine and deep learning to improve anticancer peptide prediction
Kwon, M., Jang, Y. E., Hwang, J. S., Kim, S. G., George, N. P., Basith, S., & Lee, G. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110668 · Cited by 7
Integrating Protein Language Models and Geometric Deep Learning for Peptide Toxicity Prediction
Wang, Y., Li, N., Wang, X. D., Cao, F., Xiong, S., & Wei, L. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01073 · Cited by 8
Deciphering Long-Range Effects of Mutations: An Integrated Approach Using Elastic Network Models and Protein Structure Networks
Krzesińska, K., Degn, K., Llorente, A., Giannakopoulou, E., Tiberti, M., & Papaleo, E. (2025) · Journal of Molecular Biology · doi:10.1016/j.jmb.2025.169359 · Cited by 7
ESM2_AMP: an interpretable framework for protein–protein interactions prediction and biological mechanism discovery
Sun, Y., Wang, R., Luo, Z., Tan, L., Liu, J., Li, R., Wei, D., & Zhang, Y. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf434 · Cited by 9
“Recently, protein language models (e .g., ProtT5 [13], ESM2) have become dominant, with advanced architectures such as xCAPT5 [14] and TUnA [15] achieving superior performance by capturing rich contextual and sequential dependencies from raw amino acid sequences.”
GotEnzymes2: expanding coverage of enzyme kinetics and thermal properties
Lyu, B., Wu, K., Huang, Y., Anton, M., Li, X., Viknander, S., Anwer, D., Yang, Y., Lu, D., Kerkhoven, E. J., et al. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkaf1053 · Cited by 8
Multimodal landscape of atherosclerotic plaques: A spatial omics approach with mass spectrometry imaging
Joshi, R., Tang, S. Y., Das, U. S., Boehmler, D. J., Mrčela, A., Lordan, R., Petersson, E. J., Weljie, A. M., & FitzGerald, G. A. (2025) · Analytica Chimica Acta · doi:10.1016/j.aca.2025.344723 · Cited by 7
DeepPlantAllergy: deep learning for explainable prediction of allergenicity in plant proteins
Dhouib, W., Frikha, F., Rebaï, A., & Kharrat, N. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf605 · Cited by 8
“…refinement and Gene Ontology prediction, while ESM-1B encodes biochemical properties such as hydrophobicity and polarity, biophysical properties including molecular weight and charge differences, but was also able to cluster orthologous genes without evolutionary information provided [16–18].”
Next-generation drug discovery: The AI revolution in pharmaceutical research
Bilotta, M., Rocca, R., & Alcaro, S. (2025) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2025.100149 · Cited by 11
PLysPTM-HGNN: predicting lysine PTM sites of proteins using hybrid graph neural networks
Chen, L., Yang, J., Zhou, B., & Cai, Y. (2026) · BMC Bioinformatics · doi:10.1186/s12859-025-06356-1 · Cited by 7
Root-associated protein prediction using a protein large language model and hypergraph convolutional networks
Chen, L., Xun, X., & Zhou, B. (2026) · Scientific Reports · doi:10.1038/s41598-026-35110-7 · Cited by 7
The Extreme Environment Microbiome Catalog (EEMC): a global resource for microbial diversity and antimicrobial discovery
Jiang, P., Liang, Z., Kovacevic, V., Shi, J., Milicevic, N., Wang, F., Liu, L., Liu, Y., Jiang, Y., Han, M., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-71145-0 · Cited by 8
Enhanced identification of membrane transport proteins: a hybrid approach combining ProtBERT-BFD and convolutional neural networks
Ghazikhani, H., & Butler, G. (2023) · Berichte aus der medizinischen Informatik und Bioinformatik/Journal of integrative bioinformatics · doi:10.1515/jib-2022-0055 · Cited by 7
“The loss function is computed as cross-entropy with L2 regularization on the last layer as ProtTrans project [13] in their training computations of the deep models.”
MethodInfluential
miPEPPred-FRL: A Novel Method for Predicting Plant MiRNA-Encoded Peptides Using Adaptive Feature Representation Learning
Li, H., Meng, J., Wang, Z., Tang, Y., Xia, S., Wang, Y., Qin, Z., & Luan, Y. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01020 · Cited by 7
PS4: A Next-Generation Dataset for Protein Single-Sequence Secondary Structure Prediction
Peracha, O. (2024) · BioTechniques · doi:10.2144/btn-2023-0024 · Cited by 7
Genotype sampling for deep-learning assisted experimental mapping of a combinatorially complete fitness landscape
Wagner, A. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae317 · Cited by 7
PON-Tm: A Sequence-Based Method for Prediction of Missense Mutation Effects on Protein Thermal Stability Changes
Kuang, J., Zhao, Z., Yang, Y., & Yan, W. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25158379 · Cited by 7
“…of determination ( R 2 ), root mean square error (RMSE), and mean absolute error ( MAE ). features based on an external estimator, taking into account the assigned feature weights Initially, the estimator is trained using the available features, and the importance of each feature is determined.”
Method
Advances in the Application of Protein Language Modeling for Nucleic Acid Protein Binding Site Prediction
Wang, B., & Li, W. (2024) · Genes · doi:10.3390/genes15081090 · Cited by 7
“Another notable model, ProtTrans [47] (see Table 2), shares a similar architecture to ESM2.”
BackgroundMethodInfluential
GeoNet enables the accurate prediction of protein-ligand binding sites through interpretable geometric deep learning
Han, J., Zhang, S., Guan, M., Li, Q., Gao, X., & Liu, J. (2024) · Structure · doi:10.1016/j.str.2024.10.011 · Cited by 7
An explainable few-shot learning model for the directed evolution of antimicrobial peptides
Gao, Q., Ge, L., Wang, Y., Zhu, Y., Liu, Y., Zhang, H., Huang, J., & Qin, Z. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.138272 · Cited by 8
TranP-B-site: A Transformer Enhanced Method for prediction of binding sites of Protein-protein interactions
Khan, S. H., Tayara, H., & Chong, K. T. (2025) · Measurement · doi:10.1016/j.measurement.2025.117227 · Cited by 7
How did we get there? AI applications to biological networks and sequences
Anteghini, M., Gualdi, F., & Oliva, B. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110064 · Cited by 9
Current advancement in AI-integrated drug discovery: Methods and applications
Mathur, Y., Choudhury, A., Prabha, S., Saeed, M. U., Sulaimani, M. N., Mohammad, T., & Hassan, M. I. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108642 · Cited by 8
The Power of Universal Contextualized Protein Embeddings in Cross-species Protein Function Prediction
van den Bent, I., Makrodimitris, S., & Reinders, M. J. T. (2021) · Evolutionary Bioinformatics · doi:10.1177/11769343211062608 · Cited by 9
OrganelX web server for sub-peroxisomal and sub-mitochondrial protein localization and peroxisomal target signal detection
Anteghini, M., Haja, A., dos Santos, V. A. P. M., Schomaker, L., & Saccenti, E. (2022) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.11.058 · Cited by 8
iEdgeDTA: integrated edge information and 1D graph convolutional neural networks for binding affinity prediction
Suviriyapaisal, N., & Wichadakul, D. (2023) · RSC Advances · doi:10.1039/d3ra03796g · Cited by 8
Transformer Encoder with Protein Language Model for Protein Secondary Structure Prediction
Kazm, A., Ali, A. O., & Hashim, H. B. (2024) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.6855 · Cited by 7
Homologous Pairs of Low and High Temperature Originating Proteins Spanning the Known Prokaryotic Universe
Komp, E., Alanzi, H. N., Francis, R., Vuong, C., Roberts, L., Mosallanejad, A., & Beck, D. A. C. (2023) · Scientific Data · doi:10.1038/s41597-023-02553-w · Cited by 7
FusOn-pLM: A Fusion Oncoprotein-Specific Language Model via Focused Probabilistic MaskingPreprint
Vincoff, S., Goel, S., Kholina, K., Pulugurta, R., Vure, P., & Chatterjee, P. (2024) · bioRxiv · doi:10.1101/2024.06.03.597245 · Cited by 8
Prediction of Protein Half-lives from Amino Acid Sequences by Protein Language ModelsPreprint
Sagawa, T., Kanao, E., Ogata, K., Imami, K., & Ishihama, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.10.612367 · Cited by 9
E2EDA: Protein Domain Assembly Based on End-to-End Deep Learning
(2023) · doi:10.1021/acs.jcim.3c01387 · Cited by 6
Genome-scale annotation of protein binding sites via language model and geometric deep learning
Yuan, Q., Tian, C., & Yang, Y. (2024) · eLife · doi:10.7554/elife.93695.3 · Cited by 9
Diversely evolved xibalbin variants from remipede venom inhibit potassium channels and activate PKA-II and Erk1/2 signaling
Pinheiro-Júnior, E. L., Alirahimi, E., Peigneur, S., Isensee, J., Schiffmann, S., Erkoc, P., Fürst, R., Vilcinskas, A., Senoner, T., Koludarov, I., et al. (2024) · BMC Biology · doi:10.1186/s12915-024-01955-5 · Cited by 6
NeuroPpred-SHE: An interpretable neuropeptides prediction model based on selected features from hand-crafted features and embeddings of T5 model
Wen, J., Ding, Z., Wei, Z., Xia, H., Zhang, Y., & Zhu, X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109048 · Cited by 6
The role of chromatin state in intron retention: A case study in leveraging large scale deep learning models
Daoud, A., & Ben‐Hur, A. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012755 · Cited by 10
“The same techniques have been applied to train large transformer models on large scale protein and DNA sequence datasets, yielding models that provide useful representations of biological sequences [5–7].”
Trans-MoRFs: A Disordered Protein Predictor Based on the Transformer Architecture
Meng, C., Shi, Y., Fu, X., Zou, Q., & Han, W. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3539710 · Cited by 7
Decoding the effects of mutation on protein interactions using machine learning
Wang, X., Li, A., Zhao, Y., & Peng, Y. (2025) · Biophysics Reviews · doi:10.1063/5.0249920 · Cited by 7
Enhanced prediction of antigen and antibody binding interface using ESM-2 and Bi-LSTM
Li, Q., Zhao, Y., Chordia, M. D., Xia, X., Zhang, B., & Zheng, H. (2025) · Human Immunology · doi:10.1016/j.humimm.2025.111304 · Cited by 7
Unified Deep Learning of Molecular and Protein Language Representations with T5ProtChem
Kelly, T., Xia, S., Lü, J., & Zhang, Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00051 · Cited by 6
Evaluating sequence and structural similarity metrics for predicting shared paralog functions
Dennler, O., & Ryan, C. J. (2025) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqaf051 · Cited by 6
“Recent computational advances have led to the development of new protein sequence representations, such as AlphaFold-predicted protein structures [ 25 , 26 ] and protein language model (PLM) embeddings [ 27 , 28 ].”
Influential
Unicore Enables Scalable and Accurate Phylogenetic Reconstruction with Structural Core Genes
Kim, D., Park, S., & Steinegger, M. (2025) · Genome Biology and Evolution · doi:10.1093/gbe/evaf109 · Cited by 6
Phage Host Prediction Using Deep Neural Network With Multi-Source Protein Language Models and Squeeze-and-Excitation Attention Mechanism
Gao, P., Xu, L., Bai, Y., Lin, Q., Ji, J., & Ma, L. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3582652 · Cited by 6
IECata: interpretable bilinear attention network and evidential deep learning improve the catalytic efficiency prediction of enzymes
Wang, J., Zhao, Y., Yang, Z., Ge, Y., Han, P., Liu, J., Chen, C., Zan, P., Wan, X., Bo, X., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf283 · Cited by 6
“For enzymes, enzyme sequence information was extracted by the widely used protein language model ProtT5, a transformer-based self-supervised autocoder of protein sequences [51].”
Multistage attention-based extraction and fusion of protein sequence and structural features for protein function prediction
Liu, M., Wang, S., Luo, Z., Wang, G., & Zhao, Y. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf374 · Cited by 6
Predicting antibiotic resistance genes and bacterial phenotypes based on protein language models
Wang, B., Meng, R., Li, Z., Hu, M., Wang, X., Zhao, Y., Chai, Z., Jin, Y., Yue, J., Chen, W., et al. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1628952 · Cited by 9
MoEPH: an adaptive fusion-based LLM for predicting phage-host interactions in health informatics
Chen, Q., Zhao, Z., Li, M., Song, W., Xiao, M., & Fang, M. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1634705 · Cited by 7
“In our Mixture-of-Experts (MoE) layer, a gating network dynamically computes sample-speciﬁc weights to fuse the outputs of two expert models (ProtBERT and ProT5).”
Influential
Large language models for biological sequence analysis in infectious disease research
Luo, J., Cai, X., & Li, Y. (2025) · Biosafety and Health · doi:10.1016/j.bsheal.2025.09.007 · Cited by 7
“Encoder-decoder models like ProtT5 [25] and xTrimoPGLM [11] ﬁ rst transform protein sequences into contextual embeddings by the encoder module, then generate outputs from these representations by the decoder module.”
Protein language models are accidental taxonomists
Hallee, L., Peleg, T., Rafailidis, N., & Gleghorn, J. P. (2026) · BMC Bioinformatics · doi:10.1186/s12859-026-06491-3 · Cited by 11
MPBind: a multitask protein binding site predictor using protein language models and equivariant GNNs
Wang, Y., Boadu, F., & Cheng, J. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf589 · Cited by 11
Zero-shot segmentation using embeddings from a protein language model identifies functional regions in the human proteome
Sangster, A. G., Dufault, C., Qu, H., Le, D., Forman-Kay, J. D., & Moses, A. M. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012929 · Cited by 6
“Inspired by this work, we sought to test whether a transformer-based pLM, ProtT5 [14], can perform zero-shot protein segmentation.”
Influential
Best Practices for Machine Learning-Assisted Protein Engineering
Herrera‐Rocha, F., Medina-Ortiz, D., Mauz, F., Pleiss, J., & Davari, M. D. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01983 · Cited by 12
Learning the language of phylogeny with MSA Transformer
Chen, R., Foley, G., & Mikael Bodén, A. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101445 · Cited by 7
DCGAT-DTI: dynamic cross-graph attention network for drug–target interaction prediction
Abir, A. R., Alif, M. N., Zhang, W. C., Ahmed, K. T., & Zhang, W. (2025) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf306 · Cited by 6
Integrating protein language and geometric deep learning models for enhanced vaccine antigen prediction
Zai, X., Zhao, Y., Wang, X., Leng, M., Lu, M., Yang, Y., Zhao, X., Li, R., Li, Y., Zhang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-67778-2 · Cited by 8
Disobind: A sequence-based, partner-dependent contact map and interface residue predictor for intrinsically disordered regions
Majila, K., Ullanat, V., & Viswanath, S. (2026) · Cell Systems · doi:10.1016/j.cels.2025.101486 · Cited by 8
Protein foundation models: a comprehensive survey
Xu, H., Li, L., Pan, S., Cheng, P., Wang, Y., Rong, Z., Feng, L., Huang, X., Wang, S., & Shu, W. (2026) · Science China Life Sciences · doi:10.1007/s11427-025-3147-2 · Cited by 8
Large language models for molecular design: bridging the gap between chemical syntax and biological semantics
Chen, Y., Guo, X., & Xue, W. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104634 · Cited by 7
ProtEC: A Transformer Based Deep Learning System for Accurate Annotation of Enzyme Commission Numbers
Tamir, A., Salem, M., & Yuan, J. S. (2023) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2023.3311427 · Cited by 6
Exploring the Knowledge of an Outstanding Protein to Protein Interaction Transformer
Yang, S., Cheng, P., Liu, Y., Feng, D., & Wang, S. (2024) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2024.3381825 · Cited by 6
“Prot-Trans [16] pre-trained many language models commonly used in the NLP ﬁeld and compared their performance comprehensively.”
BackgroundMethodInfluential
Rethinking the Masking Strategy for Pretraining Molecular Graphs from a Data-Centric View
Lin, W. R., & Fung, C. C. A. (2024) · ACS Omega · doi:10.1021/acsomega.3c09512 · Cited by 7
KT-AMP: Enhancing Antimicrobial Peptide Functions Prediction Through Knowledge Transfer on Protein Language Model
Liang, X., Zhao, H., & Wang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5131-0_6 · Cited by 6
Bi-SeqCNN: A Novel Light-Weight Bi-Directional CNN Architecture for Protein Function Prediction
Kumar, V., Deepak, A., Ranjan, A., & Prakash, A. (2024) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2024.3426491 · Cited by 6
PLMC: Language Model of Protein Sequences Enhances Protein Crystallization Prediction
Xiong, D., Kaicheng, U., Sun, J., & Cribbs, A. P. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00639-6 · Cited by 6
learnMSA2: deep protein multiple alignments with large language and hidden Markov models
Becker, F., & Stanke, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae381 · Cited by 6
“Large language models have been applied to proteins with empirical success (Elnaggar et al. 2021, Brandes et al. 2022, Lin et al. 2022).”
BackgroundMethodInfluential
Structure-inclusive similarity based directed GNN: a method that can control information flow to predict drug–target binding affinity
Huang, J., Sun, C. Y., Li, M., Tang, R., Xie, B., Wang, S., & Wei, J. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae563 · Cited by 6
Accurate and efficient protein embedding using multi-teacher distillation learning
Shang, J., Peng, C., Ji, Y.-X., Guan, J.-J., Cai, D.-H., Tang, X.-B., & Sun, Y.-N. (2024) · Bioinform. · doi:10.1093/bioinformatics/btae567 · Cited by 5
AAclust: k-optimized clustering for selecting redundancy-reduced sets of amino acid scales
Breimann, S., & Frishman, D. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae165 · Cited by 6
Ab-Amy 2.0: Predicting light chain amyloidogenic risk of therapeutic antibodies based on antibody language model
Zhou, Y., Liu, W., Luo, C., Huang, Z., Savini, G. S. M., Zhao, L., Wang, R., & Huang, J. (2024) · Methods · doi:10.1016/j.ymeth.2024.11.005 · Cited by 6
Evaluation of predictions of disordered binding regions in the CAID2 experiment
Zhang, F., & Kurgan, L. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.009 · Cited by 7
Paying attention to the SARS-CoV-2 dialect : a deep neural network approach to predicting novel protein mutations
Elkin, M. E., & Xingquan Zhu, A. (2025) · Communications Biology · doi:10.1038/s42003-024-07262-7 · Cited by 6
Subcellular Enrichment Patterns of New Genes in Drosophila Evolution
Dong, C., Xia, S., Zhang, L., Arsala, D., Fang, C., Tan, S., Clark, A. G., & Long, M. (2025) · Molecular Biology and Evolution · doi:10.1093/molbev/msaf038 · Cited by 5
ATP-Pred: Prediction of Protein-ATP Binding Residues via Fusion of Residue-Level Embeddings and Kolmogorov–Arnold Network
Zhang, L., & Liu, T. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00016 · Cited by 7
A Deep Learning-Based Approach for Predicting Michaelis Constants from Enzymatic Reactions
Li, Y., & Wang, K. (2025) · Applied Sciences · doi:10.3390/app15074017 · Cited by 6
Language models learn to represent antigenic properties of human influenza A(H3) virus
Durazzi, F., Koopmans, M., Fouchier, R. A. M., & Remondini, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-03275-2 · Cited by 9
Lightweight ProteinUnet2 network for protein secondary structure prediction: a step towards proper evaluation
Stąpor, K., Kotowski, K., Smolarczyk, T., & Roterman, I. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04623-z · Cited by 7
“It includes also the results on one of the best protein language models ProtT5XL-U50 [19] for reference.”
Background
Predicting the specific substrate for transmembrane transport proteins using BERT language model
Ataei, S., & Butler, G. (2022) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb55180.2022.9863051 · Cited by 5
“Recent work on ProtTrans [30] studied six Transformer-based architectures to predict the secondary structure and subcellular localization.”
Method
Enzyme Activity Prediction of Sequence Variants on Novel Substrates using Improved Substrate Encodings and Convolutional Pooling.
Xu, Z.-Q. J., Wu, J., Song, Y. S., & Mahadevan, R. (2022) · PubMed · Cited by 6
TooT-BERT-C: A study on discriminating ion channels from membrane proteins based on the primary sequence’s contextual representation from BERT models
Ghazikhani, H., & Butler, G. (2022) · International Conference on Bioinformatics Research and Applications (ICBRA) · doi:10.1145/3569192.3569196 · Cited by 6
PhosBoost: Improved phosphorylation prediction recall using gradient boosting and protein language models
Poretsky, E., Andorf, C. M., & Sen, T. Z. (2023) · Plant Direct · doi:10.1002/pld3.554 · Cited by 6
“By pre-training on vast numbers of protein sequences, pLMs learned inherent properties encoded within protein sequences, revolutionizing multiple fields of protein research (Bordin et al., 2023; Elnaggar et al., 2021; Ofer et al., 2021; Rives et al., 2021).”
BackgroundMethod
Large language models for biomolecular analysis: From methods to applications
Feng, R., Zhang, C., & Zhang, Y. (2024) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2024.117540 · Cited by 7
A novel data augmentation approach for influenza A subtype prediction based on HA proteins
Sohrabi, M. A., Zare‐Mirakabad, F., Ghidary, S. S., Saadat, M., & Sadegh‐Zadeh, S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108316 · Cited by 6
TooT-BERT-T: A BERT Approach on Discriminating Transport Proteins from Non-transport Proteins
Ghazikhani, H., & Butler, G. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-17024-9_1 · Cited by 6
PRO-LDM: Protein Sequence Generation with a Conditional Latent Diffusion ModelPreprint
Zhang, S., Jiang, Z., Huang, R., Mo, S., Zhu, L.-T., Li, P., Zhang, Z., Pan, E., Chen, X., Long, Y.-F., et al. (2024) · bioRxiv · doi:10.1101/2023.08.22.554145 · Cited by 11
Accurate Prediction of Antibody Function and Structure Using Bio-Inspired Antibody Language ModelPreprint
Jing, H., Gao, Z., Xu, S. Y., Shen, T., Peng, Z., He, S., You, T., Ye, S., Lin, W., & Sun, S. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.30.555473 · Cited by 6
MPEK: a multi-task learning based on pre-trained language model for predicting enzymatic reaction kinetic parametersPreprint
Jiang, H., Wang, J., Yang, Z., Chen, C., Ge, Y., Bao, S., Wan, X., Ding, J., & Wang, L. (2024) · Research Square · doi:10.21203/rs.3.rs-3916417/v1 · Cited by 6
ProteusAI: An Open-Source and User-Friendly Platform for Machine Learning-Guided Protein Design and EngineeringPreprint
Funk, J., Machado, L., Bradley, S. A., Napiorkowska, M., Gallegos-Dextre, R., Pashkova, L., Madsen, N. G., Webel, H. E., Phaneuf, P. V., Jenkins, T. P., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.01.616114 · Cited by 7
Benchmarking gene embeddings from sequence, expression, network, and text models for functional prediction tasksPreprint
Zhong, J., Li, L., Dannenfelser, R., & Yao, V. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.29.635607 · Cited by 9
Transformers meets neoantigen detection: a systematic literature review
Machaca, V., Goyzueta, V., Cruz, M. G., Sejje, E., Pilco, L. M., López, J., & Túpac, Y. (2024) · Journal of Integrative Bioinformatics · doi:10.1515/jib-2023-0043 · Cited by 5
PEPerMINT: peptide abundance imputation in mass spectrometry-based proteomics using graph neural networks
Pietz, T., Gupta, S., Schlaffner, C. N., Ahmed, S., Steen, H., Renard, B. Y., & Baum, K. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae389 · Cited by 5
“Also, amino acid sequence information is available for missing proteins and peptides, and language models pre-trained on amino acid sequences have shown good performance on a variety of protein-related tasks (Elnaggar et al. 2021).”
BackgroundMethodInfluential
Post-processing enhances protein secondary structure prediction with second order deep learning and embeddings
Chatzimiltis, S., Agathocleous, M., Promponas, V. J., & Christodoulou, C. C. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.022 · Cited by 8
“Those embeddings were extracted from the ProtBert model [42,43].”
Storage access optimization for efficient GPU-centric information retrieval
Shrestha, S., Gautam, A., & Reddy, N. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07118-9 · Cited by 5

Authors: Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
Affiliations: Technical University of Munich (DE)
Jiangnan University (CN)
Google (United States) (US)
Nvidia (United States) (US)
NVIDIA, Santa Clara, CA, USA
Seoul National University (KR)
Oak Ridge National Laboratory (US)
ORCID: Elnaggar, Ahmed
Heinzinger, Michael
Dallago, Christian
Rehawi, Ghalia
Wang, Yu
Gibbs, Tom
Feher, Tamas
Steinegger, Martin
Bhowmik, Debsindhu
Rost, Burkhard
Venue: IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2022-10-01
Volume: 44
Issue: 10
Pages: 7112-7127
ISSN: 0162-8828 (ISSN-L)
2160-9292
1939-3539
DOI: 10.1109/tpami.2021.3095381
OpenAlex: W3177500196
Cited by: 2,329
References: 101
Citations by year: 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
FWCI: 71.82
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/legalcode (vor)
creativecommons.org/licenses/by/4.0/legalcode (am)
Locations: IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
PubMed (publishedVersion)
mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Seoul National University Open Repository (Seoul National University) (submittedVersion)
Funders: National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
New Faculty Startup Fund and the Creative-Pioneering Researchers Program
Seoul National University
Google
Oak Ridge National Laboratory
U.S. Department of Energy (DE-AC05-00OR22725)
TensorFlow Research Cloud
Topics: Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
Keywords: protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
MeSH terms: Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks