Cited by 2,329Machine Learning in BioinformaticsProtein Structure and Dynamicsvaccines and immunoinformatics approaches

The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.

ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Open access

Elnaggar, Ahmed, Heinzinger, Michael, Dallago, Christian, Rehawi, Ghalia, Wang, Yu, Jones, Llion, Gibbs, Tom, Feher, Tamas, Angerer, Christoph, Steinegger, Martin, Bhowmik, Debsindhu, Rost, Burkhard·2022·IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI 10.1109/tpami.2021.3095381

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.

Integrity signals

Registry

Crossref · 10.1109/tpami.2021.3095381

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
The Protein Data Bank
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Untitled work
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213

Flagged first, then by citations

Show all 101 references

2 of the 101 references couldn’t be matched to a known paper.

Research works printed but not linked: 5

john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
limited, 2019, Press release announcing Supercomputer Fugaku
rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
10.1110/ps.03128904 · DOI

Sorted by citations

Items 501–600 of 1,007

Uncovering the Interaction Interface Between Harpin (Hpa1) and Rice Aquaporin (OsPIP1;3) Through Protein–Protein Docking: An In Silico Approach
Patoliya, J., Thaker, K., Rabadiya, K., Patel, D., Jain, N. K., & Joshi, R. G. (2023) · Molecular Biotechnology · doi:10.1007/s12033-023-00690-6 · Cited by 16
Protein Fitness Prediction Is Impacted by the Interplay of Language Models, Ensemble Learning, and Sampling Methods
Mardikoraem, M., & Woldring, D. (2023) · Pharmaceutics · doi:10.3390/pharmaceutics15051337 · Cited by 16
“This has been tested with different NLP techniques, varying model parameters, and clustering sizes for databases used and resulted in a wide array of language-based protein representations [30,66,68,69].”
Method
AttSec: protein secondary structure prediction by capturing local patterns from attention map
(2023) · doi:10.1186/s12859-023-05310-3 · Cited by 16
“Table 6 compares our model with the two best-performing single sequence-based models, ProtT5-XL-U50 and NetsurfP-3.”
MethodInfluential
AggBERT: Best in Class Prediction of Hexapeptide Amyloidogenesis with a Semi-Supervised ProtBERT Model
Perez, R. M., Li, X., Giannakoulias, S. G., & Petersson, E. J. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00817 · Cited by 16
Generative artificial intelligence performs rudimentary structural biology modeling
Ille, A. M., Markosian, C., Burley, S. K., Mathews, M. B., Pasqualini, R., & Arap, W. (2024) · Scientific Reports · doi:10.1038/s41598-024-69021-2 · Cited by 16
Are genomic language models all you need? Exploring genomic language models on protein downstream tasks
Boshar, S., Trop, E., de Almeida, B. P., Copoiu, L., & Pierrot, T. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae529 · Cited by 16
MVSF-AB: accurate antibody–antigen binding affinity prediction via multi-view sequence feature learning
Li, M., Shi, Y., Hu, S., Hu, S., Guo, P., Wan, W., Zhang, L. Y., Pan, S., Li, J., Sun, L., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae579 · Cited by 16
Protein engineering in the deep learning era
Zhou, B., Tan, Y., Hu, Y., Zheng, L., Zhong, B., & Hong, L. (2024) · mLife · doi:10.1002/mlf2.12157 · Cited by 18
Accelerating protein engineering with fitness landscape modelling and reinforcement learning
Sun, H., He, L., Deng, P., Liu, G., Zhao, Z., Jiang, Y., Cao, C., Ju, F., Wu, L., Liu, H., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01103-w · Cited by 19
“…TEM-1 [33]) and reporter proteins (GFP [34]) of high interest in synthetic biology, disease-related proteins (Amyloid beta precursor protein APP [35]), signaling proteins (PSD-95 [36], GRB2 [36] and YAP1 [37]), RNA binding proteins (Pab1 [38]), and protein binding proteins (GB1 [39]).”
PepMLM: Target Sequence-Conditioned Generation of Therapeutic Peptide Binders via Span Masked Language ModelingPreprint
Chen, T., Dumas, M., Watson, R., Vincoff, S., Peng, C., Zhao, L., Hong, L., Pertsemlidis, S., Shaepers-Cheu, M., Wang, T. Z., et al. (2023) · PubMed · doi:10.48550/arxiv.2310.03842 · Cited by 24
ProteinCLIP: enhancing protein language models with natural languagePreprint
Wu, K., Chang, H. Y., & Zou, J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.14.594226 · Cited by 19
Causal modeling of gene effects from regulators to programs to traits: integration of genetic associations and Perturb-seqPreprint
Ota, M., Spence, J. P., Zeng, T., Dann, E., Marson, A., & Pritchard, J. K. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.22.634424 · Cited by 16
ProtSpace: A Tool for Visualizing Protein Space
Senoner, T., Olenyi, T., Heinzinger, M., Spannagl, A., Bouras, G. S., Rost, B., & Koludarov, I. (2025) · Journal of Molecular Biology · doi:10.1016/j.jmb.2025.168940 · Cited by 17
Twenty years of advances in prediction of nucleic acid-binding residues in protein sequences
Basu, S., Jing, Y., Kihara, D., & Kurgan, L. (2024) · Briefings Bioinform. · doi:10.1093/bib/bbaf016 · Cited by 15
FusOn-pLM: a fusion oncoprotein-specific language model via adjusted rate masking
Vincoff, S., Goel, S., Kholina, K., Pulugurta, R., Vure, P., & Chatterjee, P. (2025) · Nature Communications · doi:10.1038/s41467-025-56745-6 · Cited by 20
ProBASS—a language model with sequence and structural features for predicting the effect of mutations on binding affinity
Gurusinghe, S. N. S., Wu, Y., DeGrado, W. F., & Shifman, J. M. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf270 · Cited by 15
Hypergraph-based semantic and topological self-supervised learning for brain disease diagnosis
Han, X., Lei, M., & Li, J. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111921 · Cited by 17
Phytopathogen Effector Biology in the Burgeoning AI Era
Jones, D., & Raffaele, S. (2025) · Annual Review of Phytopathology · doi:10.1146/annurev-phyto-121823-081033 · Cited by 17
AmpHGT: expanding prediction of antimicrobial activity in peptides containing non-canonical amino acids using multi-view constrained heterogeneous graph transformer
He, Y., Song, X., Wan, H., & Zhao, X. (2025) · BMC Biology · doi:10.1186/s12915-025-02253-4 · Cited by 17
Computational methods and data resources for predicting tumor neoantigens
Zhao, X., Wei, L., & Zhang, X. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf302 · Cited by 16
Recent Advances in Machine Learning‐Assisted Design and Development of Polymer Materials
Ma, L., Li, W., Yuan, J., Zhu, J., Wu, Y., He, H., & Pan, X. (2025) · Macromolecular Rapid Communications · doi:10.1002/marc.202500361 · Cited by 17
Top-DTI: integrating topological deep learning and large language models for drug–target interaction prediction
Talo, M., & Bozdag, S. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf183 · Cited by 16
Artificial intelligence-based methods for protein structure prediction: a survey
Zhan, Z., Hong, J., Li, J.-Y., Wang, C., He, L., Xu, Z., & Zhang, J. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11325-4 · Cited by 17
Critical Assessment of Protein Intrinsic Disorder Round 3 ‐ Predicting Disorder in the Era of Protein Language Models
Mehdiabadi, M., Conte, A. D., Nugnes, M. V., Aspromonte, M. C., Tosatto, S. C. E., & Piovesan, D. (2025) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.70045 · Cited by 20
Flexynesis: A deep learning toolkit for bulk multi-omics data integration for precision oncology and beyond
Uyar, B., Savchyn, T., Nilchi, A. N., Sarıgün, A., Wurmus, R., Shaik, M. M., Grüning, B. A., Franke, V., & Akalin, A. (2025) · Nature Communications · doi:10.1038/s41467-025-63688-5 · Cited by 20
Artificial intelligence for nanomedicine
Song, X., Gao, X., Wang, H., Yu, F., Qin, M., Li, Y., Liu, Y., Feng, W., Zhou, C., Chukavin, N. N., et al. (2025) · Science China Chemistry · doi:10.1007/s11426-025-2942-5 · Cited by 16
GlobDB: a comprehensive species-dereplicated microbial genome resource
Speth, D. R., Pullen, N., Aroney, S. T. N., Coltman, B. L., Osvatic, J., Woodcroft, B. J., Rattei, T., & Wagner, M. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf280 · Cited by 15
Improving Enzyme Fitness with Machine Learning
Patsch, D., & Buller, R. (2023) · CHIMIA International Journal for Chemistry · doi:10.2533/chimia.2023.116 · Cited by 14
Protein Function Prediction With Functional and Topological Knowledge of Gene Ontology
Zhao, Y., Yang, Z., Hong, Y., Yang, Y., Wang, L., Zhang, Y., Lin, H., & Wang, J. (2023) · IEEE Transactions on NanoBioscience · doi:10.1109/tnb.2023.3278033 · Cited by 15
FCMSTrans: Accurate Prediction of Disease-Associated nsSNPs by Utilizing Multiscale Convolution and Deep Feature Combination within a Transformer Framework
Zhang, M., Gong, C., Ge, F., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02025 · Cited by 15
DeepP450: Predicting Human P450 Activities of Small Molecules by Integrating Pretrained Protein Language Model and Molecular Representation
Chang, J., Fan, X., & Tian, B. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00115 · Cited by 15
Protein function prediction through multi-view multi-label latent tensor reconstruction
Armah-Sekum, R. E., Szedmák, S., & Rousu, J. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05789-4 · Cited by 16
A comprehensive review of protein-centric predictors for biomolecular interactions: from proteins to nucleic acids and beyond
Jia, P., Zhang, F., Wu, C., & Li, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae162 · Cited by 15
“It is worth noting that EGRET [122] generates protein embeddings using the self-supervised protein language model ProtTrans [128].”
Cancer mutational signatures identification in clinical assays using neural embedding-based representations
Yaacov, A., Cohen, G. B., Landau, J., Hope, T., Simon, I., & Rosenberg, S. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2024.101608 · Cited by 18
“22 The BERT-based transformer methodology was applied to protein and DNA sequences by ProtBert 23 and DNABERT.”
Machine learning for predicting protein properties: A comprehensive review
Wang, Y., Zhang, Y., Zhan, X., He, Y., Yang, Y., Cheng, L., & Alghazzawi, D. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128103 · Cited by 15
Prediction of human O- linked glycosylation sites using stacked generalization and embeddings from pre-trained protein language model
Pakhrin, S. C., Chauhan, N., Khan, S., Upadhyaya, J., Beck, M. R., & Blanco, E. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae643 · Cited by 15
Harnessing pre-trained models for accurate prediction of protein-ligand binding affinity
Li, J., & Gong, X. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06064-w · Cited by 14
Protein language models for predicting drug–target interactions: Novel approaches, emerging methods, and future directions
Ünlü, A., Ulusoy, E., Yiğit, M. G., Darcan, M., & Doğan, T. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103017 · Cited by 15
Aggregating residue-level protein language model embeddings with optimal transport
Naderializadeh, N., & Singh, R. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf060 · Cited by 17
Exploring the Protein Sequence Space with Global Generative ModelsPreprint
Romero‐Romero, S., Lindner, S., & Ferruz, N. (2023) · Cold Spring Harbor Perspectives in Biology · doi:10.1101/cshperspect.a041471 · Cited by 18
Predicting Metal-binding Proteins and Structures Through Integration of Evolutionary-scale and Physics-based Modeling
(2025) · doi:10.1016/j.jmb.2025.168962 · Cited by 17
Computational design of novel Cas9 PAM-interacting domains using evolution-based modelling and structural quality assessment
Malbranke, C., Rostain, W., Depardieu, F., Cocco, S., Monasson, R., & Bikard, D. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011621 · Cited by 15
“Unlike evolutionary-based methods that rely solely on sequence information [12, 13], structural modeling of protein families incorporates physical and chemical considerations.”
Method
A Knowledge‐Guided Graph Learning Approach Bridging Phenotype‐ and Target‐Based Drug Discovery
Ye, Q., Zeng, Y., Jiang, L., Kang, Y., Pan, P., Chen, J., Deng, Y., Zhao, H., He, S., Hou, T., et al. (2025) · Advanced Science · doi:10.1002/advs.202412402 · Cited by 16
Prediction and design of thermostable proteins with a desired melting temperature
Tijare, P. A., Kumar, N., & Raghava, G. P. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-98667-9 · Cited by 16
Designing a nitrogen-efficient cold-tolerant maize for modern agricultural systems
Ojeda‐Rivera, J. O., Barnes, A. C., Ainsworth, E. A., Angelovici, R., Basso, B., Brindisi, L. J., Brooks, M. D., Busch, W., Buttelmann, G. L., Castellano, M. J., et al. (2025) · The Plant Cell · doi:10.1093/plcell/koaf139 · Cited by 15
De Novo Design of Multiple Microplastic-Binding Peptides with a Protein Language Model-Guided Generative Adversarial Network
Wang, S.-Y., Bergman, M. T., Hall, C. K., & You, F.-Q. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01401 · Cited by 12
Computational function prediction of bacteria and phage proteins
Grigson, S. R., Bouras, G. S., Dutilh, B. E., Olson, R., & Edwards, R. A. (2025) · Microbiology and Molecular Biology Reviews · doi:10.1128/mmbr.00022-25 · Cited by 18
Recent Advances in the Prediction of Subcellular Localization of Proteins and Related Topics
Nakai, K., & Wei, L. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.910531 · Cited by 15
“Six was constructed based on ProtTrans and a significant improvement of its performance is reported (Teufel et al., 2022).”
Background
Variational autoencoder for design of synthetic viral vector serotypes
Lyu, S., Sowlati‐Hashjin, S., & Garton, M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-023-00787-2 · Cited by 27
DeepPepPI: A deep cross-dependent framework with information sharing mechanism for predicting plant peptide-protein interactions
Wang, Z., Meng, J., Dai, Q., Li, H., Xia, S., Yang, R., & Luan, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124168 · Cited by 14
ProFun-SOM: Protein Function Prediction for Specific Ontology Based on Multiple Sequence Alignment Reconstruction
Shao, J., Chen, J., & Liu, B. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3419250 · Cited by 14
Protein-peptide binding residue prediction based on protein language models and cross-attention mechanism
Hu, J., Chen, K., Rao, B., Ni, J.-Y., Thafar, M. A., Albaradei, S., & Arif, M. (2024) · Analytical Biochemistry · doi:10.1016/j.ab.2024.115637 · Cited by 14
In the twilight zone of protein sequence homology: do protein language models learn protein structure?
Kabir, A., Moldwin, A., Bromberg, Y., & Shehu, A. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae119 · Cited by 13
“While advancing remote homology prediction is an active area of research with an increasing number of models and methodologies (Hamamsy et al. 2023, Kaminski et al. 2023, Kilinc et al. 2023, Johnson et al. 2024, Liu et al. 2024) we focus here on the following SOTA PLMs: TAPE-BERT (Rao et al. 2019), Protein-BERT (Brand…”
Influential
DeepNeoAG: Neoantigen epitope prediction from melanoma antigens using a synergistic deep learning model combining protein language models and multi-window scanning convolutional neural networks
Chuang, C.-C., Liu, Y., & Ou, Y. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.136252 · Cited by 17
dsAMP and dsAMPGAN: Deep Learning Networks for Antimicrobial Peptides Recognition and Generation
Zhao, M., Zhang, Y., Wang, M., & Luyan, Z. (2024) · Antibiotics · doi:10.3390/antibiotics13100948 · Cited by 14
“In addition, the influence of pre-trained models from natural language processing, such as Protrans [25] and word2vec [26], is expanding into protein sequence analysis.”
BackgroundMethod
Beyond digital twins: the role of foundation models in enhancing the interpretability of multiomics modalities in precision medicine
Alsaedi, S. B., Gao, X., & Gojobori, T. (2025) · FEBS Open Bio · doi:10.1002/2211-5463.70003 · Cited by 19
LocPro: A deep learning-based prediction of protein subcellular localization for promoting multi-directional pharmaceutical research
Zhang, Y., Zheng, L., You, N., Hu, W., Jiang, W., Lu, M., Xu, H., Dai, H., Fu, T., & Zhou, Y. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101255 · Cited by 15
“Forexample,proteinlanguage modelssuchasESM2[19],ProteinBERT[20],andProtT5[21],through unsupervised pretraining on large-scale protein sequence datasets, have demonstratedthe ability to capture latent structural, functional, and evolutionary information embedded in amino acid sequences [22].”
PBertKla: a protein large language model for predicting human lysine lactylation sites
Lai, H., Luo, D., Yang, M., Zhu, T., Yang, H., Luo, X., Wei, Y., Xie, S., Hong, F., Shu, K., et al. (2025) · BMC Biology · doi:10.1186/s12915-025-02202-1 · Cited by 16
Large language models transform biological research: from architecture to utilization
Wang, T., & Luo, Z. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4466-3 · Cited by 14
Protein-Protein Interaction Prediction is Achievable with Large Language ModelsPreprint
Hallee, L., & Gleghorn, J. P. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.07.544109 · Cited by 17
Structure-enabled enzyme function prediction unveils elusive terpenoid biosynthesis in archaeaPreprint
Samusevich, R., Akmeşe, S. M., Hebra, T., Bushuiev, R., Engst, M., Kulhánek, J., Bushuiev, A., Smith, J. D., Čalounová, T., Smrčková, H., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.29.577750 · Cited by 18
ProteinGLUE multi-task benchmark suite for self-supervised protein modeling
Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., & Feenstra, K. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-19608-4 · Cited by 11
“Although the CB513 dataset 55 has been called redundant and outdated 11 , it is commonly used to evaluate model quality on the prediction of secondary structure in eight classes as it is sufficiently difficult to avoid ceiling effects.”
BackgroundMethodInfluential
Whole-genome prediction of bacterial pathogenic capacity on novel bacteria using protein language models with PathogenFinder2
Ferrer Florensa, A., Almagro Armenteros, J. J., Kaas, R. S., Clausen, P. T. L. C., Nielsen, H., Rost, B., & Aarestrup, F. M. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag129 · Cited by 50
Language models for protein design
Lee, J. S., Abdin, O., & Kim, P. M. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103027 · Cited by 15
Accurate and Rapid Prediction of Protein p K a : Protein Language Models Reveal the Sequence–p K a Relationship
Xu, S., & Onoda, A. (2025) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.4c01288 · Cited by 13
Foundation models in plant molecular biology: advances, challenges, and future directions
Xu, F., Wu, T., Qian, C., Wang, X., & Yan, J. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1611992 · Cited by 14
Tokenized and continuous embedding compressions of protein sequence and structure
Lu, A. X., Yan, W., Yang, K. K., Gligorijević, V., Cho, K., Abbeel, P., Bonneau, R., & Frey, N. C. (2025) · Patterns · doi:10.1016/j.patter.2025.101289 · Cited by 15
Leveraging artificial intelligence and machine learning for unraveling pathogenesis and advancing precision medicine in autoimmune diseases
Cao, C., Zhao, W., Guo, J., Wang, Z., Lü, A., & Liang, C. C. (2025) · The Innovation Medicine · doi:10.59717/j.xinn-med.2025.100154 · Cited by 14
Conditional generation of real antigen-specific T cell receptor sequences
Karthikeyan, D., Bennett, S., Reynolds, A. G., Vincent, B. G., & Rubinsteyn, A. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01096-6 · Cited by 16
TCRdesign: an antigen-specific generative language model for de novo design of T-cell receptors
Li, X., Yang, Q., Xu, L., Dong, W., Wang, K., Dong, S., Wang, W., Luo, G., Zhang, X., Yang, T., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf691 · Cited by 14
RNAGEN: A Generative Adversarial Network‐Based Model to Generate Synthetic RNA Sequences to Target Proteins
Özden, F., Barazandeh, S., Akboğa, D., Tabrizi, S. S., Şeker, U. Ö. Ş., & Çiçek, A. E. (2026) · ChemBioChem · doi:10.1002/cbic.202500445 · Cited by 13
Running ahead of evolution—AI-based simulation for predicting future high-risk SARS-CoV-2 variants
Chen, J., Nie, Z., Wang, Y., Wang, K., Xu, F., Hu, Z., Zheng, B., Wang, Z., Song, G., Zhang, J., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231188077 · Cited by 12
Exploration of DPP-IV Inhibitory Peptide Design Rules Assisted by the Deep Learning Pipeline That Identifies the Restriction Enzyme Cutting Site
Guan, C., Luo, J., Li, S., Tan, Z. L., Wang, Y., Chen, H., Yamamoto, N., Zhang, C., Lu, Y., Chen, J., et al. (2023) · ACS Omega · doi:10.1021/acsomega.3c05571 · Cited by 13
ProtT5 and random forests-based viscosity prediction method for therapeutic mAbs
Hao, X., & Fan, L. (2024) · European Journal of Pharmaceutical Sciences · doi:10.1016/j.ejps.2024.106705 · Cited by 13
Predicting Protein Functions Based on Heterogeneous Graph Attention Technique
Zhao, Y., Yang, Z., Wang, L., Zhang, Y., Lin, H., & Wang, J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3357834 · Cited by 13
Multilevel Attention Network with Semi-supervised Domain Adaptation for Drug-Target Prediction
Xie, Z., Tu, S., & Xu, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27786 · Cited by 14
HemoDL: Hemolytic peptides prediction by double ensemble engines from Rich sequence-derived and transformer-enhanced information
Yang, S., & Xu, P. (2024) · Analytical Biochemistry · doi:10.1016/j.ab.2024.115523 · Cited by 15
Fungtion: A Server for Predicting and Visualizing Fungal Effector Proteins
Li, J., Ren, J., Dai, W., Stubenrauch, C. J., Finn, R. D., & Wang, J. (2024) · Journal of Molecular Biology · doi:10.1016/j.jmb.2024.168613 · Cited by 13
Protein language models enable prediction of polyreactivity of monospecific, bispecific, and heavy-chain-only antibodies
Yu, X., Vangjeli, K., Prakash, A., Chhaya, M., Stanley, S. J., Cohen, N. D., & Huang, L. (2024) · Antibody Therapeutics · doi:10.1093/abt/tbae012 · Cited by 13
SOFB is a comprehensive ensemble deep learning approach for elucidating and characterizing protein-nucleic-acid-binding residues
Zhang, B., Hou, Z., Yang, Y., Wong, K., Zhu, H., & Li, X. (2024) · Communications Biology · doi:10.1038/s42003-024-06332-0 · Cited by 13
Heterogeneous biomedical entity representation learning for gene–disease association prediction
Meng, Z., Liu, S., Liang, S., Jani, B., & Meng, Z. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae380 · Cited by 13
“ESM-2b, a protein language model [12, 23], excels in generating comprehensive embeddings from protein sequences by learning from extensive evolutionary data.”
BackgroundMethodInfluential
PreAlgPro: Prediction of allergenic proteins with pre-trained protein language model and efficient neutral network
Zhang, L., & Liu, T. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.135762 · Cited by 14
Current computational tools for protein lysine acylation site prediction
Qin, Z., Ren, H., Zhao, P., Wang, K., Liu, H., Miao, C., Du, Y., Li, J., Wu, L., & Chen, Z. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae469 · Cited by 13
ParaAntiProt provides paratope prediction using antibody and protein language models
Kalemati, M., Noroozi, A., Shahbakhsh, A., & Koohi, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-80940-y · Cited by 15
iMFP-LG: Identify Novel Multi-functional Peptides Using Protein Language Models and Graph-based Deep Learning
Luo, J., Zhao, K., Chen, J., Yang, C., Qu, F., Liu, Y., Jin, X., Yan, K., Zhang, Y., & Liu, B. (2024) · Genomics Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzae084 · Cited by 21
Comprehensive prediction and analysis of human protein essentiality based on a pretrained large language model
Kang, B., Fan, R., Cui, C., & Cui, Q. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00733-1 · Cited by 13
Recent Applications of Artificial Intelligence in Discovery of New Antibacterial Agents
Bagdad, Y., & Miteva, M. A. (2024) · Advances and Applications in Bioinformatics and Chemistry · doi:10.2147/aabc.s484321 · Cited by 16
MVGNN-PPIS: A novel multi-view graph neural network for protein-protein interaction sites prediction based on Alphafold3-predicted structures and transfer learning
Lu, M., Wei, L., & Wu, R. (2025) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2025.140096 · Cited by 15
Barlow Twins deep neural network for advanced 1D drug–target interaction prediction
Schuh, M. G., Boldini, D., Bohne, A. I., & Sieber, S. A. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-00952-2 · Cited by 13
MultiKD-DTA: Enhancing Drug-Target Affinity Prediction Through Multiscale Feature Extraction
Hu, R., Ge, R., Deng, G., Fan, J., Tang, B., & Wang, C. (2025) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-025-00697-4 · Cited by 13
Comparative Assessment of Protein Large Language Models for Enzyme Commission Number Prediction
Capela, J., Zimmermann‐Kogadeeva, M., van Dijk, A. D. J., de Ridder, D., Dias, Ó., & Rocha, M. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06081-9 · Cited by 13
Unveiling the evolution of antimicrobial peptides in gut microbes via foundation-model-powered framework
Li, W., Huang, B., Guo, M., Zeng, Z., Cai, T., Feng, L., Zhang, X., Guo, L., Jiang, X., Yin, Y., et al. (2025) · Cell Reports · doi:10.1016/j.celrep.2025.115773 · Cited by 13
Leveraging Natural Language Processing models to decode the dark proteome across the Animal Tree of LifePreprint
Redondo, G. I. M., Perez-Canales, F. M., Fernández, J. M., Barrios-Núñez, I., Vázquez-Valls, M., Cases, I., Rojas, A. M., & Fernández, R. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.02.28.582465 · Cited by 13
VirulentHunter: deep learning-based virulence factor predictor illuminates pathogenicity in diverse microbial contexts
Chen, C., Xu, Y., Ouyang, J., Xiong, X., Łabaj, P. P., Chmielarczyk, A., Różańska, A., Zhang, H., Liu, K., Shi, T., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf271 · Cited by 15
“Recent advances in protein language models, such as ESM2 [19], ProteinBERT [20] and ProtT5 [21] have been pre-trained and our previous studies has demonstrated their ability to extract complex sequence–structure function relationships [22, 23].”
Generative prediction of real-world prevalent SARS-CoV-2 mutation with in silico virus evolution
Liu, X., Nie, Z., Si, H.-R., Shen, X., Liu, Y., Huang, X., Dong, T., Xu, F., Ren, Z., Zhou, P., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf276 · Cited by 13
Machine learning-assisted rational design and evolution of novel signal peptides in Yarrowia lipolytica
Wu, Z., Chen, W., Hong, Y., Wang, Y., & Xu, P. (2025) · Synthetic and Systems Biotechnology · doi:10.1016/j.synbio.2025.07.008 · Cited by 12
“The ProtTrans embed features can implicitly capture biophysical and structural information from amino acid sequences [35].”
GS-DTI: a graph-structure-aware framework leveraging large language models for drug–target interaction prediction
Yu, Q., Zhou, C., Jiang, J., Shi, X., & Li, Y. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf445 · Cited by 14
Paying attention to attention: High attention sites as indicators of protein family and function in language models
Nayar, G., Tartici, A., & Altman, R. B. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013424 · Cited by 12
Evaluating Pretrained Protein Language Model Embeddings as Proxies for Functional Similarity
Shaw, R., Love, S. D., & McWhite, C. D. (2025) · Journal of Molecular Evolution · doi:10.1007/s00239-025-10282-4 · Cited by 12
Cell-penetrating peptides for therapeutic applications: emerging design strategies and future directions
Kim, E., Lee, Y., Ryu, J., Kim, H., Park, J., & Lee, J. (2026) · Organic & Biomolecular Chemistry · doi:10.1039/d6ob00146g · Cited by 12

Authors: Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
Affiliations: Technical University of Munich (DE)
Jiangnan University (CN)
Google (United States) (US)
Nvidia (United States) (US)
NVIDIA, Santa Clara, CA, USA
Seoul National University (KR)
Oak Ridge National Laboratory (US)
ORCID: Elnaggar, Ahmed
Heinzinger, Michael
Dallago, Christian
Rehawi, Ghalia
Wang, Yu
Gibbs, Tom
Feher, Tamas
Steinegger, Martin
Bhowmik, Debsindhu
Rost, Burkhard
Venue: IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2022-10-01
Volume: 44
Issue: 10
Pages: 7112-7127
ISSN: 0162-8828 (ISSN-L)
2160-9292
1939-3539
DOI: 10.1109/tpami.2021.3095381
OpenAlex: W3177500196
Cited by: 2,329
References: 101
Citations by year: 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
FWCI: 71.82
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/legalcode (vor)
creativecommons.org/licenses/by/4.0/legalcode (am)
Locations: IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
PubMed (publishedVersion)
mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Seoul National University Open Repository (Seoul National University) (submittedVersion)
Funders: National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
New Faculty Startup Fund and the Creative-Pioneering Researchers Program
Seoul National University
Google
Oak Ridge National Laboratory
U.S. Department of Energy (DE-AC05-00OR22725)
TensorFlow Research Cloud
Topics: Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
Keywords: protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
MeSH terms: Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks