Cited by 2,329Machine Learning in BioinformaticsProtein Structure and Dynamicsvaccines and immunoinformatics approaches

The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.

ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Open access

Elnaggar, Ahmed, Heinzinger, Michael, Dallago, Christian, Rehawi, Ghalia, Wang, Yu, Jones, Llion, Gibbs, Tom, Feher, Tamas, Angerer, Christoph, Steinegger, Martin, Bhowmik, Debsindhu, Rost, Burkhard·2022·IEEE Transactions on Pattern Analysis and Machine Intelligence

DOI 10.1109/tpami.2021.3095381

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.

Integrity signals

Registry

Crossref · 10.1109/tpami.2021.3095381

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
The Protein Data Bank
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Untitled work
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213

Show all 101 references

Research works printed but not linked: 5

john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
limited, 2019, Press release announcing Supercomputer Fugaku
rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
10.1110/ps.03128904 · DOI

Sorted by citations

Items 601–700 of 1,007

Improving antibody optimization ability of generative adversarial network through large language model
Zhao, W., Luo, X., Tong, F., Zheng, X., Li, J., Zhao, G., & Zhao, D. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.041 · Cited by 13
“ProtBERT [34] has been trained using the BERT [35] architecture and the UniRef100 and BDF100 datasets.”
Method
E-pRSA: Embeddings Improve the Prediction of Residue Relative Solvent Accessibility in Protein Sequence
Manfredi, M., Savojardo, C., Martelli, P. L., & Casadio, R. (2024) · Journal of Molecular Biology · doi:10.1016/j.jmb.2024.168494 · Cited by 12
Recent Advancements in Subcellular Proteomics: Growing Impact of Organellar Protein Niches on the Understanding of Cell Biology
Bhushan, V., & Nita‐Lazar, A. (2024) · Journal of Proteome Research · doi:10.1021/acs.jproteome.3c00839 · Cited by 15
A systematic analysis of regression models for protein engineering
Richard, M., Kæstel‐Hansen, J., Groth, P. M., Bartels, S., Salomon, J., Tian, P., Hatzakis, N. S., & Boomsma, W. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012061 · Cited by 21
Exploring the sequence-function space of microbial fucosidases
Gascueña, A. M., Wu, H., Wang, R., Owen, C. D., Hernando, P. J., Monaco, S., Penner, M., Xing, K., Gall, G. L., Gardner, R. A., et al. (2024) · Communications Chemistry · doi:10.1038/s42004-024-01212-4 · Cited by 11
Advancing biofoundry development: strategies and challenges
Gupta, A., Lee, S., Sung, B. H., Lee, D., Cho, B., Kim, D., & Kim, H. (2024) · Biotechnology and Bioprocess Engineering · doi:10.1007/s12257-024-00121-6 · Cited by 12
Fine-tuning protein embeddings for functional similarity evaluation
Dickson, A. M., & Mofrad, M. R. K. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae445 · Cited by 15
Protein–Protein Interaction Prediction Model Based on ProtBert-BiGRU-Attention
Gao, Q., Zhang, C., Li, M., & Yu, T. (2024) · Journal of Computational Biology · doi:10.1089/cmb.2023.0297 · Cited by 12
Leveraging multiple data types for improved compound-kinase bioactivity prediction
Theisen, R., Wang, T., Ravikumar, B., Rahman, R., & Cichońska, A. (2024) · Nature Communications · doi:10.1038/s41467-024-52055-5 · Cited by 13
DeepAIP: Deep learning for anti-inflammatory peptide prediction using pre-trained protein language model features based on contextual self-attention network
Zhu, L., Yang, Q., & Yang, S. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.136172 · Cited by 12
Uncovering the taste features: Applying machine learning and molecular docking approaches to predict umami taste intensity of peptides
Ashikhmina, M. S., Zenkin, A. M., Pantiukhin, I. S., Litvak, I. G., Nesterov, P. V., Dutta, K., Shityakov, S., Nosonovsky, M., Korablev-Dyson, M. A., Orlova, O. Y., et al. (2024) · Food Bioscience · doi:10.1016/j.fbio.2024.105358 · Cited by 12
MIPPIS: protein–protein interaction site prediction network with multi-information fusion
Wang, S., Dong, K., Liang, D., Zhang, Y., Li, X., & Song, T. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05964-7 · Cited by 12
Advances in Zero‐Shot Prediction‐Guided Enzyme Engineering Using Machine Learning
Liu, C., Wu, J., Chen, Y., Liu, Y., Zheng, Y., Liu, L., & Zhao, J. B. (2024) · ChemCatChem · doi:10.1002/cctc.202401542 · Cited by 15
UniAMP: enhancing AMP prediction using deep neural networks with inferred information of peptides
Chen, Z., Ji, C., Xu, W., Gao, J., Huang, J., Xu, H., Qian, G., & Huang, J. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06033-3 · Cited by 14
ProCeSa: Contrast-Enhanced Structure-Aware Network for Thermostability Prediction with Protein Language Models
Zhou, F., Zhang, S., Zhang, H., & Liu, J. K. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01752 · Cited by 15
PUNCH2: Explore the strategy for intrinsically disordered protein predictor
Meng, D., & Pollastri, G. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319208 · Cited by 14
Recent advances and future trends for protein–small molecule interaction predictions with protein language models
Kroll, A., & Rousset, Y. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103070 · Cited by 12
Machine-Guided Dual-Objective Protein Engineering for Deimmunization and Therapeutic Functions
Wolfsberg, E., Paul, J.-S., Tycko, J., Chen, B., Bassik, M. C., Bintu, L., Alizadeh, A. A., & Gao, X. J. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101299 · Cited by 12
SPRoBERTa: protein embedding learning with local fragment modeling
Wu, L., Yin, C., Zhu, J., Wu, Z., He, L., Xia, Y., Xie, S., Qin, T., & Liu, T. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac401 · Cited by 12
“• LM-GVP [40] is an extended model of GVP [41], which prepends ProtBERT [17] before GVP to utilize the sequence representation.”
MethodInfluential
Prediction of polyreactive and nonspecific single-chain fragment variables through structural biochemical features and protein language-based descriptors
Lim, H., & No, K. T. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-05010-4 · Cited by 12
AbBERT: Learning Antibody Humanness via Masked Language ModelingPreprint
Vashchenko, D., Nguyen, S., Gonçalves, A. R., da Silva, F. L., Petersen, B. K., Desautels, T., & Faissol, D. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.02.502236 · Cited by 13
“ProtTrans (1) introduces auto-regressive and auto-encoder models trained on data from UniRef (2) and BFD (3).”
Method
Diverse Genomic Embedding Benchmark for functional evaluation across the tree of lifePreprint
West-Roberts, J. A., Kravitz, J., Jha, N., Cornman, A., & Hwang, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.10.602933 · Cited by 13
Protein domain embeddings for fast and accurate similarity searchPreprint
Iovino, B. G., Tang, H., & Ye, Y. (2024) · Genome Research · doi:10.1101/gr.279127.124 · Cited by 12
Decoding the Molecular Language of Proteins with EvollaPreprint
Zhou, X., Han, C., Zhang, Y., Du, H., Tian, J., Su, J., Liu, R., Zhuang, K., Jiang, S., Gitter, A., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.05.630192 · Cited by 16
The Dayhoff Atlas: scaling sequence diversity for improved protein generationPreprint
Yang, K., Alamdari, S., Lee, A. J., Kaymak-Loveless, K., Char, S., Brixi, G., Domingo-Enrich, C., Wang, C., Lyu, S., Fusi, N., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.21.665991 · Cited by 13
PlantCAD2: a DNA foundation model for interpreting genomes across flowering plantsPreprint
Zhai, J., Gokaslan, A., Hsu, S., Chen, S.-P., Liu, Z.-Y., Marroquin, E., Czech, E., Cannon, B. R., Berthel, A., Romay, M. C., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.27.672609 · Cited by 12
Knot or not? Identifying unknotted proteins in knotted families with sequence‐based Machine Learning model
Sikora, M., Klimentova, E., Uchal, D., Sramkova, D., Perlinska, A. P., Nguyen, M. L., Korpacz, M., Malinowska, R., Nowakowski, S., Rubach, P., et al. (2024) · Protein Science · doi:10.1002/pro.4998 · Cited by 13
ResUbiNet: A Novel Deep Learning Architecture for Ubiquitination Site Prediction
Duan, Z., Liang, Y., Xin, X. L., Ma, W., & Hu, M. Y. (2024) · Current Genomics · doi:10.2174/0113892029331751240820111158 · Cited by 11
BPFun: a deep learning framework for bioactive peptide function prediction using multi-label strategy by transformer-driven and sequence rich intrinsic information
Zhu, L., Sun, H., & Yang, S. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06190-5 · Cited by 11
Boosting the predictive power of protein representations with a corpus of text annotations
Duan, H., Skreta, M., Cotta, L., Rajaonson, E. M., Dhawan, N., Aspuru‐Guzik, A., & Maddison, C. J. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01088-6 · Cited by 11
Graph neural network integrated with pretrained protein language model for predicting human–virus protein–protein interactions
Jiang, L., Yang, X., Guo, X., Li, D., Li, J., Wuchty, S., Shi, W., & Zhang, Z. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf461 · Cited by 11
“We also compared the results of various pretrained models (SaProt [23], ESM2 [19], and TAPE [24]) as alternatives to ProtT5 [21] for encoding protein amino acid residues.”
Influential
NbBench: benchmarking language models for comprehensive nanobody tasks
Zhang, Y., & Tsuda, K. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae20ec · Cited by 16
Enzyme property prediction using artificial intelligence
Yuan, L., Shafaei, S., & Zhao, H. (2025) · Current Opinion in Chemical Engineering · doi:10.1016/j.coche.2025.101208 · Cited by 14
PlasticEnz: An integrated database and screening tool combining homology and machine learning to identify plastic-degrading enzymes in meta-omics datasets
Krzynowek, A., Snoeks, J., & Faust, K. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013892 · Cited by 12
“( 4 ) Sequences that pass this homology screen are embedded using ProtBERT [39] to generate contextualized feature vectors.”
PeptideNet: An Integrative Deep Learning Framework for Predicting Diverse Bioactive Peptides Using Protein Language Model Embeddings
Zahid, H., Maryam, Chong, K. T., & Tayara, H. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c02885 · Cited by 11
PepBERT: Lightweight language models for bioactive peptide representation
Du, Z., Caragea, D., Guo, X., & Li, Y. (2026) · Future Foods · doi:10.1016/j.fufo.2026.100999 · Cited by 14
IFF : Identifying key residues in intrinsically disordered regions of proteins using machine learning
Ho, W., Huang, H., & Huang, J. (2023) · Protein Science · doi:10.1002/pro.4739 · Cited by 11
LncPNdeep: A long non-coding RNA classifier based on large language model with peptide and nucleotide embedding
Dai, Z., Deng, F., & Sung, H. H. (2026) · Non-coding RNA Research · doi:10.1016/j.ncrna.2026.06.004 · Cited by 12
“LLM-based architectures have shown state-of-the-art results in diverse tasks such as sequence classification and question answering [30,31], while self-supervised and semi-supervised pre-training allow generalized features to be learned from raw data [32 – 34].”
Unveiling Recent Trends in Biomedical Artificial Intelligence Research: Analysis of Top-Cited Papers
Glicksberg, B. S., & Klang, E. (2024) · Applied Sciences · doi:10.3390/app14020785 · Cited by 12
MucLiPred: Multi-Level Contrastive Learning for Predicting Nucleic Acid Binding Residues of Proteins
Zhang, J., Wang, R., & Wei, L. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01471 · Cited by 12
A BERT-based approach for identifying anti-inflammatory peptides using sequence information
Xu, T., Wang, Q., Yang, Z., & Ying, J. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32951 · Cited by 13
Predicting protein functions using positive-unlabeled ranking with ontology-based priors
Zhapa-Camacho, F., Tang, Z., Kulmanov, M., & Hoehndorf, R. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae237 · Cited by 13
PreDBP-PLMs: Prediction of DNA-binding proteins based on pre-trained protein language models and convolutional neural networks
Qi, D., Chen, S., & Liu, T. (2024) · Analytical Biochemistry · doi:10.1016/j.ab.2024.115603 · Cited by 11
Computational analysis of pathogen-host interactome for fast and low-risk in-silico drug repurposing in emerging viral threats like Mpox
Paul, D., Saha, S., Basu, S., & Chakraborti, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-69617-8 · Cited by 11
DTVF: A User-Friendly Tool for Virulence Factor Prediction Based on ProtT5 and Deep Transfer Learning Models
Sun, J., Yin, H., Ju, C., Wang, Y., & Yang, Z. (2024) · Genes · doi:10.3390/genes15091170 · Cited by 11
GGN-GO: geometric graph networks for predicting protein function by multi-scale structure features
Mi, J., Wang, H., Li, J., Sun, J., Li, C., Wan, J., Zeng, Y., & Gao, J. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae559 · Cited by 12
“For sequence features, we used the protein language models ESM2 [12] and Prot-Trans [13].”
ProtChat: An AI Multi-Agent for Automated Protein Analysis Leveraging GPT-4 and Protein Language Model
Huang, H., Shi, X., Lei, H., Hu, F., & Cai, Y. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01345 · Cited by 11
Applications of Transformers in Computational Chemistry: Recent Progress and Prospects
Wang, R., Ji, Y., Li, Y., & Lee, S. (2024) · The Journal of Physical Chemistry Letters · doi:10.1021/acs.jpclett.4c03128 · Cited by 13
TransConv: convolution-infused transformer for protein secondary structure prediction
Das, S., Ghosh, S., & Jana, N. D. (2025) · Journal of Molecular Modeling · doi:10.1007/s00894-024-06259-7 · Cited by 12
Prediction of inhibitory peptides against E.coli with desired MIC value
Bajiya, N., Kumar, N., & Raghava, G. P. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-86638-z · Cited by 15
CaLMPhosKAN: prediction of general phosphorylation sites in proteins via fusion of codon aware embeddings with amino acid aware embeddings and wavelet-based Kolmogorov–Arnold network
Pratyush, P., Carrier, C., Pokharel, S., Ismail, H. D., Chaudhari, M., & KC, D. B. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf124 · Cited by 11
“To the best of our knowledge, this is the first work to utilize codon-aware embeddings in the PTM prediction problem and residue-level task in general.”
TransBind allows precise detection of DNA-binding proteins and residues using language models and deep learning
Tahmid, M. T., Hasan, A. K. M. M., & Bayzid, M. S. (2025) · Communications Biology · doi:10.1038/s42003-025-07534-w · Cited by 11
Deep learning methods for protein representation and function prediction: A comprehensive overview
Wang, M., Nie, Z., He, Y., Vasilakos, A. V., Cheng, Q. S., & Ren, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110977 · Cited by 11
Fine-tuning protein language models to understand the functional impact of missense variants
Saadat, A., & Fellay, J. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.05.022 · Cited by 16
DACMF-DTI: Dual attention embedded cross-modality fusion for drug-target interaction prediction
Wei, Z., Xiao, Z., Li, C., Wang, M., & Tang, C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114063 · Cited by 13
A large language model for predicting neurotoxic peptides and neurotoxins
Rathore, A. S., Jain, S., Choudhury, S., & Raghava, G. P. S. (2025) · Protein Science · doi:10.1002/pro.70200 · Cited by 9
Integration of pre-trained protein language models with equivariant graph neural networks for peptide toxicity prediction
Jiao, S., Ye, X., Sakurai, T., Zou, Q., Han, W., & Zhan, C. (2025) · BMC Biology · doi:10.1186/s12915-025-02329-1 · Cited by 14
Tokenization and deep learning architectures in genomics: A comprehensive review
Testagrose, C., & Boucher, C. A. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.07.038 · Cited by 13
Painting Peptides With Antimicrobial Potency Through Deep Reinforcement Learning
Dong, R., Cao, Q., & Song, C. (2025) · Advanced Science · doi:10.1002/advs.202506332 · Cited by 11
Predictive and therapeutic applications of protein language models
Furui, K., Sakano, K., & Ohue, M. (2025) · Allergology International · doi:10.1016/j.alit.2025.08.004 · Cited by 12
Functional alignment of protein language models via reinforcement learning
Blalock, N., Seshadri, S., Nakamura, K., Babbar, A., Fahlberg, S. A., Kulkarni, A., & Romero, P. A. (2026) · Nature Communications · doi:10.1038/s41467-026-77557-2 · Cited by 19
Does protein pretrained language model facilitate the prediction of protein–ligand interaction?
Zhang, W., Hu, F., Wang, L., & Yin, P. (2023) · Methods · doi:10.1016/j.ymeth.2023.08.016 · Cited by 11
AraPathogen2.0: An Improved Prediction of Plant–Pathogen Protein–Protein Interactions Empowered by the Natural Language Processing Technique
Lei, C., Zhou, K., Zheng, J., Zhao, M., Huang, Y., He, H., Yang, S., & Zhang, Z. (2023) · Journal of Proteome Research · doi:10.1021/acs.jproteome.3c00364 · Cited by 11
Improving PTM Site Prediction by Coupling of Multi-Granularity Structure and Multi-Scale Sequence Representation
Li, Z., Li, M., Zhu, L., & Zhang, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27770 · Cited by 12
FoldHSphere: deep hyperspherical embeddings for protein fold recognition
Villegas-Morcillo, A., Sanchez, V., & Gomez, A. M. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04419-7 · Cited by 11
ProtPlat: an efficient pre-training platform for protein classification based on FastText
Jin, Y., & Yang, Y. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04604-2 · Cited by 11
“As mentioned in Background, ProtTrans uses various transformer models, which were trained on a supercomputer with 936 nodes (total 5616 GPUs) and one TPU Pod (V3-512 or V3-1024).”
BackgroundMethodInfluential
Unexplored regions of the protein sequence-structure map revealed at scale by a library of foldtuned language modelsPreprint
Subramanian, A. M., Martinez, Z. A., Lourenço, A. L., Yuan, S. C., Liu, S., Wang, T.-Y., Chou, T., & Thomson, M. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.22.573145 · Cited by 10
PROTGOAT : Improved automated protein function predictions using Protein Language ModelsPreprint
Chua, Z. M., Rajesh, A., Sinha, S., & Adams, P. D. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.01.587572 · Cited by 12
Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systems
Kramer, S., Cerrato, M., Brugger, J., Džeroski, S., & King, R. (2023) · arXiv · doi:10.48550/arxiv.2305.02251 · Cited by 10
Pre-trained Maldi Transformers improve MALDI-TOF MS-based prediction
(2025) · doi:10.1016/j.compbiomed.2025.109695 · Cited by 17
Fine-tuning protein language models unlocks the potential of underrepresented viral proteomes
Sawhney, R., Ferrell, B. D., Dejean, T., Schreiber, Z., Harrigan, W., Polson, S. W., Wommack, K. E., & Belcaid, M. (2025) · PeerJ · doi:10.7717/peerj.19919 · Cited by 14
“Recent studies suggest this approach has been successful in natural language processing and proteomics ( Ding & Steinhardt, 2024 ; Elnaggar et al., 2021 ; Rao et al., 2019 ; Gu et al., 2021 ; Brandes et al., 2022 ; Rives et al., 2021 ; Howard & Ruder, 2018 ; Devlin et al., 2019 ; Houlsby et al.,…”
A survey on large language models in biology and chemistry
Ashyrmamatov, I., Gwak, S. J., Jin, S.-Y., Jun, I., Ucak, U. V., Lee, J. Y., & Lee, J. (2025) · Experimental & Molecular Medicine · doi:10.1038/s12276-025-01583-1 · Cited by 15
Machine learning for enzyme catalytic activity: current progress and future horizons
Qiu, S., Saeed, H., Leonard, W., Li, F., & Yang, A. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag002 · Cited by 13
LBCE-XGB: A XGBoost Model for Predicting Linear B-Cell Epitopes Based on BERT Embeddings
Liu, Y., Liu, Y., Wang, S., & Zhu, X. (2023) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-023-00549-z · Cited by 10
EmbedCaps-DBP: Predicting DNA-Binding Proteins Using Protein Sequence Embedding and Capsule Network
Mursalim, M. K. N., Mengko, T. L., Hertadi, R., Purwarianti, A., & Susanty, M. (2023) · IEEE Access · doi:10.1109/access.2023.3328960 · Cited by 10
PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning
Ramachandran, A., Lumetta, S. S., & Chen, D. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011790 · Cited by 14
Experimental and computational approaches for membrane protein insertion and topology determination
Duart, G., Graña‐Montes, R., Pastor‐Cantizano, N., & Mingarro, I. (2024) · Methods · doi:10.1016/j.ymeth.2024.03.012 · Cited by 11
AbLEF: antibody language ensemble fusion for thermodynamically empowered property predictions
Rollins, Z. A., Widatalla, T., Waight, A. B., Cheng, A. C., & Metwally, E. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae268 · Cited by 9
Generative design of compounds with desired potency from target protein sequences using a multimodal biochemical language model
Chen, H., & Bajorath, J. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-024-00852-x · Cited by 11
Algorithmic Issues, Challenges, and Theoretical Concerns of ChatGPT
Patil, P., Kulkarni, K., & Sharma, P. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-6824-4.ch003 · Cited by 11
Molecular sequence classification using efficient kernel based embedding
Ali, S., Ali, T. E., Murad, T., Mansoor, H., & Patterson, M. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121100 · Cited by 11
A Dual-Modality Complex-Valued Fusion Method for Predicting Side Effects of Drug-Drug Interactions Based on Graph Neural Network
Kang, C., Zhang, H., & Yin, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3422673 · Cited by 10
PANDA-3D: protein function prediction based on AlphaFold models
Zhao, C., Liu, T., & Wang, Z. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae094 · Cited by 11
“Littmann et al. ( 21 ) found that predicting GO terms based on proximity of embeddings from language models SeqVec ( 22 ) or ProtBert ( 23 ) outperformed naïve sequence-based transfer.”
AI-accelerated therapeutic antibody development: practical insights
Santuari, L., Salvy, M. B., Xénarios, I., & Arpat, A. B. (2024) · Frontiers in Drug Discovery · doi:10.3389/fddsv.2024.1447867 · Cited by 10
Comprehensive Assessment of BERT-Based Methods for Predicting Antimicrobial Peptides
Gao, W. J., Zhao, J. W., Gui, J., Wang, Z., Chen, J., & Yue, Z. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00507 · Cited by 13
Porter 6: Protein Secondary Structure Prediction by Leveraging Pre-Trained Language Models (PLMs)
Alanazi, W. T., Meng, D., & Pollastri, G. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms26010130 · Cited by 10
Modern machine learning methods for protein property prediction.
Dosajh, A., Agrawal, P., Chatterjee, P., & Priyakumar, U. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102990 · Cited by 7
MutualDTA: An Interpretable Drug–Target Affinity Prediction Model Leveraging Pretrained Models and Mutual Attention
Yuan, Y., Chen, S., Hu, R., & Wang, X. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01893 · Cited by 11
TransHLA: a Hybrid Transformer model for HLA-presented epitope detection
Lu, T., Wang, X., Nie, W., Huo, M., & Li, S. (2025) · GigaScience · doi:10.1093/gigascience/giaf008 · Cited by 10
Effectidor II: a pan-genomic AI-based algorithm for the prediction of type III secretion system effectors
Wagner, N., Baumer, E., Lyubman, I., Shimony, Y., Bracha, N., Martins, L., Potnis, N., Chang, J. H., Teper, D., Koebnik, R., et al. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf272 · Cited by 10
Protein Sequence Analysis landscape: A Systematic Review of Task Types, Databases, Datasets, Word Embeddings Methods, and Language Models
Asim, M. N., Asif, T., Hassan, F., & Dengel, A. (2025) · Database · doi:10.1093/database/baaf027 · Cited by 11
Rag2Mol: structure-based drug design based on retrieval augmented generation
Zhang, P., Peng, X., Han, R., Chen, T., & Ma, J. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf265 · Cited by 12
BertADP: a fine-tuned protein language model for anti-diabetic peptide prediction
Xie, X., Wu, C., Qi, Y., Liu, S., Huang, J., Lv, H., Dao, F., & Lin, H. (2025) · BMC Biology · doi:10.1186/s12915-025-02312-w · Cited by 11
In silico prediction of variant effects: promises and limitations for precision plant breeding
Sendrowski, J., Bataillon, T. M., & Ramstein, G. P. (2025) · Theoretical and Applied Genetics · doi:10.1007/s00122-025-04973-1 · Cited by 12
ProtAlign-ARG: antibiotic resistance gene characterization integrating protein language models and alignment-based scoring
Ahmed, S., Emon, M. I., Moumi, N. A., Huang, L., Zhou, D., Vikesland, P. J., Pruden, A. J., & Zhang, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-14545-4 · Cited by 10
NetStart 2.0: prediction of eukaryotic translation initiation sites using a protein language model
Nielsen, L. S., Pedersen, A. G., Winther, O., & Nielsen, H. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06220-2 · Cited by 10
Transfer learning on protein language models improves antimicrobial peptide classification
Georgoulis, E., Zervou, M. A., & Pantazis, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-21223-y · Cited by 13
Active learning-guided optimization of cell-free biosensors for lead testing in drinking water
Wang, B. M., Chiang, N., Ekas, H. M., Brown, D. M., Dildine, G., Lucci, T. J., Feng, S., Bly, V., Gaillard, J., Lucks, J. B., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-66964-6 · Cited by 13
rzMLP-DTA: gMLP network with ReZero for sequence-based drug-target affinity prediction
Qiu, Z., Jiao, Q., Wang, Y., Chen, C., Zhu, D., & Cui, X. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669276 · Cited by 9
OncoRTT: Predicting novel oncology-related therapeutic targets using BERT embeddings and omics features
Thafar, M. A., Albaradei, S., Uludağ, M., Alshahrani, M., Gojobori, T., Essack, M., & Gao, X. (2023) · Frontiers in Genetics · doi:10.3389/fgene.2023.1139626 · Cited by 11
“ProtTrans trained two autoregressive models (Transformer-X and XLNet) and four auto-encoder models (BERT, Albert, Electra, and T5) using more than two thousand million protein sequences, making it the state-of-the-art method to encode the protein sequence with the highest quality embeddings.”
BackgroundMethodInfluential

Authors: Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
Affiliations: Technical University of Munich (DE)
Jiangnan University (CN)
Google (United States) (US)
Nvidia (United States) (US)
NVIDIA, Santa Clara, CA, USA
Seoul National University (KR)
Oak Ridge National Laboratory (US)
ORCID: Elnaggar, Ahmed
Heinzinger, Michael
Dallago, Christian
Rehawi, Ghalia
Wang, Yu
Gibbs, Tom
Feher, Tamas
Steinegger, Martin
Bhowmik, Debsindhu
Rost, Burkhard
Venue: IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2022-10-01
Volume: 44
Issue: 10
Pages: 7112-7127
ISSN: 0162-8828 (ISSN-L)
2160-9292
1939-3539
DOI: 10.1109/tpami.2021.3095381
OpenAlex: W3177500196
Cited by: 2,329
References: 101
Citations by year: 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
FWCI: 71.82
Type: journal article
Language: en
Open access: hybrid
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/legalcode (vor)
creativecommons.org/licenses/by/4.0/legalcode (am)
Locations: IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
PubMed (publishedVersion)
mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Seoul National University Open Repository (Seoul National University) (submittedVersion)
Funders: National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
New Faculty Startup Fund and the Creative-Pioneering Researchers Program
Seoul National University
Google
Oak Ridge National Laboratory
U.S. Department of Energy (DE-AC05-00OR22725)
TensorFlow Research Cloud
Topics: Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
Keywords: protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
MeSH terms: Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks