The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Open accessPublished version of a preprint
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Fine-tuning has not been extensively explored in previous work5, possibly due to its demanding computational requirements.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“Such models were then applied to protein sequence: the BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model [58] for protein sequence prediction.”
From Computational and artificial intelligence-based methods for antibody development (2023) · cites this paper as [58]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Flagged first, then by citations
2 of the 101 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
- limited, 2019, Press release announcing Supercomputer Fugaku
- rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
- armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
- 10.1110/ps.03128904 · DOI
Cited by
Sorted by citations
Items 901–1,000 of 1,007
A multimodal deep learning framework for enzyme turnover prediction with missing modality
Sun, X., Wang, Y. G., & Shen, Y. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110348 · Cited by 6
Explainability of Protein Deep Learning Models
Fazel, Z., de Souza, C. P. E., Golding, G. B., & Ilie, L. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26115255 · Cited by 5
“Deep learning approaches have provided unprecedented insights into protein structure prediction, function annotation, and interaction dynamics, enabling researchers to explore the complexities of biological systems with remarkable precision and efficiency [1–4].”
Ou, Y., Chen, Q., Zhong, N., & Du, Z. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf285 · Cited by 5
“ProtPhage integrates ProtT5 [17] embeddings and an asymmetric loss (ASL) function, [18] forming a synergistic solution to the challenges of sequence divergence and class imbalance.”
ABPDB: A Database of Antibacterial Peptides
Zhou, X., Liu, G., Han, C., & Lv, J. Z. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3582844 · Cited by 5
Pratyush, P., Pokharel, S., Schulze, S., Bramer, L., Newman, R. H., & KC, D. B. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_19 · Cited by 6
CLAPE: Protein–Ligand Binding Site Prediction via Protein Language Models
Liu, Y., & Tian, B. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_18 · Cited by 5
Xie, C., Wei, Y., Luo, X., Yang, H., Lai, H., Dao, F., Feng, J., & Lv, H. (2025) · BMC Biology · doi:10.1186/s12915-025-02314-8 · Cited by 5
LoRA-DR-suite: adapted embeddings predict intrinsic and soft disorder from protein sequences
Lombardi, G., Seoane, B., & Carbone, A. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf185 · Cited by 4
“In this study, we employed well-established PLMs as pre-trained models, including two versions of the Evolutionary Scale Model (ESM2) with 35M and 650M parameters (Lin et al. 2023), ProtT5 from the ProtTrans suite (Elnaggar et al. 2022), and Ankh (Elnaggar et al. 2023).”
Yan, Z., Ge, F., Zhang, Y., He, Y., Liu, Y., Zhu, Y., Song, J., & Yu, D. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00756 · Cited by 5
Jin, Q., Wu, A., Wei, L., Cui, H. S., Xuan, P., Feng, X., & Su, R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111743 · Cited by 5
Allofusion: Allosteric Site Prediction Based on Language Models and Multi-Feature Fusion
Huang, J., Guo, D., Liu, Y., Wang, Y., & Meng-ya, L. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01033 · Cited by 5
Wang, F., Chu, J., Shen, L., & Chang, S. (2025) · BMC Biology · doi:10.1186/s12915-025-02356-y · Cited by 5
Biological Sequence Representation Methods and Recent Advances: A Review
Zhang, H., Shi, Y., Wang, Y., Xu, Y., Li, K., Im, S., & Han, Y. D. (2025) · Biology · doi:10.3390/biology14091137 · Cited by 7
“Their high computational complexity, stemming from multi-objective optimization and large-scale datasets, requires substantial resources, limiting accessibility [22,23,89,94,96].”
Commensal gut bacteria employ de-chelatase HmuS to harvest iron from heme
Nath, A. K., da Silva, R. R., Gauvin, C. C., Akpoto, E., Adedoyin, V., Dlakić, M., Lawrence, C. M., & DuBois, J. L. (2025) · The EMBO Journal · doi:10.1038/s44318-025-00563-5 · Cited by 6
Isaksen, A., Nachtigall, P. G., Araya, R. A., Beh, J. H., Robinson, S. D., Hansen, T. F., & Undheim, E. A. B. (2025) · Genome biology · doi:10.1186/s13059-025-03777-2 · Cited by 5
Hu, F., Zeng, W., & Peng, S. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf524 · Cited by 5
“Similarly, ULDNA [24] incorporates unsupervised PLM derived from various database sources (including ESM2,ProtTrans [25],and ESM-MSA [26]) with an LSTM-attention network, which enhances the accuracy of predicting protein-DNA binding sites.”
A protein dynamics–based deep learning model enhances predictions of fitness and epistasis
Huynh, N., Kazan, I. C., Lu, J., Kolbaba‐Kartchner, B., Mills, J. H., & Ozkan, S. B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2502444122 · Cited by 7
Spicer, R., Raychawdhary, N., Danwada, S., Udomprasert, P., Seals, C., & Bhattacharya, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-23319-x · Cited by 5
del Río, J. S., Talavera, A., Fernández‐Pozo, N., Veredas, F. J., & Claros, M. G. (2025) · Journal of Experimental Botany · doi:10.1093/jxb/eraf498 · Cited by 6
Reimagining computational macromolecular modeling: AI-driven approaches
Özdemir, E. S., Jang, H., Nussinov, R., Keskin, Ö., & Gürsoy, A. (2025) · Biophysical Journal · doi:10.1016/j.bpj.2025.11.028 · Cited by 6
Protein embeddings and local alignments
Malec, J., Golding, G. B., & Ilie, L. (2026) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.12.002 · Cited by 5
“The advent of protein embeddings such as ProtT5 [10], ESM2 [40], or Ankh [9], has revolutionized bioinformatics by providing contex tual representations that capture functional and evolutionary patterns.”
Hidden in protein sequences: Predicting cryptic binding sites
Škrhák, V., Gamouh, H., & Hoksza, D. (2025) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3765612.3767221 · Cited by 4
The DNA dialect: a comprehensive guide to pretrained genomic language models
Veiner, M., & Supek, F. (2026) · Molecular Systems Biology · doi:10.1038/s44320-025-00184-4 · Cited by 10
Generative AI for synthetic biology: Designing biological parts, circuits, and genomes
Kim, N., Carluccio, G. D., Zhang, K., & Collins, J. J. (2026) · Cell Systems · doi:10.1016/j.cels.2026.101533 · Cited by 5
Molecular Modeling and machine learning for predicting high-concentration antibody viscosity
Baizhigitova, D., Wu, I.-E., Kalejaye, L. A., & Lai, P. (2026) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2026.115839 · Cited by 5
Liang, Y., & Wang, C. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c02745 · Cited by 5
Artificial intelligence in antibiotic discovery: Applications, challenges, and future outlook
Xu, J. B., Li, C., Wang, X., Peleg, A. Y., LI, F., Song, J., & de la Fuente‐Núñez, C. (2026) · Cell Biomaterials · doi:10.1016/j.celbio.2026.100445 · Cited by 9
Toward life with a 19–amino acid alphabet through generative artificial intelligence design
Liu, L., Rochereau, C., Kozlov, S., Urtecho, G., Liu, X., Zhao, J., Wang, J., Huang, Y., Qu, Y., Zhang, Z., et al. (2026) · Science · doi:10.1126/science.aeb5171 · Cited by 7
Towards the explainability of protein language models
Hunklinger, A., & Ferruz, N. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01232-w · Cited by 12
“Before pLMs, often most high-performing approaches used multiple sequence alignment (MSA) for various tasks, from remote homology detection 33 to structure prediction. pLMs were originally proposed as single-sequence alternatives that could compensate for sequences with few homologues 34,35 .”
Contemporary data-driven innovations in peptide-based therapeutic design
Priyadarsinee, L., Kungurtsev, V., Kumar, V., Chatterjee, B., Sastry, G. N., & Murugan, N. A. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag220 · Cited by 5
Translating functional molecular knowledge into crop-breeding success
Ramstein, G. P., Zhai, J., Buckler, E. S., Tenaillon, M. I., & Jannink, J. (2026) · Nature Reviews Genetics · doi:10.1038/s41576-026-00968-w · Cited by 7
Artificial intelligence tools for enzyme engineering and metabolic engineering
Volk, M., Pillalamarri, V. K., Guo, R., Liu, G., & Zhao, H. (2026) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2026.103522 · Cited by 5
PlantCAD2: A DNA foundation model for interpreting genomes across flowering plants
Zhai, J., Gokaslan, A., Hsu, S.-K., Chen, S.-P., Liu, Z.-Y., Marroquin, E., Czech, E., Cannon, B. R., Berthel, A., Romay, M. C., et al. (2026) · Cell Genomics · doi:10.1016/j.xgen.2026.101329 · Cited by 25
PseAAC2Vec protein encoding for TCR protein sequence classification
Tayebi, Z., Ali, S., Murad, T., Khan, I., & Patterson, M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.107956 · Cited by 6
Clark, J. D., Mi, X., Mitchell, D. A., & Shukla, D. (2024) · Digital Discovery · doi:10.1039/d4dd00170b · Cited by 8
ThermoFinder: A sequence-based thermophilic proteins prediction framework
Yu, H., & Luo, X. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.132469 · Cited by 5
Dong, N., Kampffmeyer, M., Su, H., & Xing, E. P. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111855 · Cited by 5
Qi, D., & Liu, T. (2024) · Biochimica et Biophysica Acta (BBA) - General Subjects · doi:10.1016/j.bbagen.2024.130721 · Cited by 5
GPS-pPLM: A Language Model for Prediction of Prokaryotic Phosphorylation Sites
Zhang, C., Tang, D., Han, C., Gou, Y., Chen, M., Huang, X., Liu, D., Zhao, M., Xiao, L., Xiao, Q., et al. (2024) · Cells · doi:10.3390/cells13221854 · Cited by 5
“Meanwhile, PhosphoLingo incorporated the contextual features generated from various PLMs, such as Evolutionary-Scale Modeling (ESM) [34] and ProtT5 [32], to predict eukaryotic p-sites [35].”
Le, V., Malik, M. S., Lin, Y.-J., Liu, Y., Chang, Y., & Ou, Y. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109541 · Cited by 5
Benchmarking recent computational tools for DNA-binding protein identification
Luo, X., Yi, A. S., HuiKai, A. L., Ong, T. J., Wong, L., & Rahman, C. R. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae634 · Cited by 6
Lv, Z., Wei, M., Pei, H., Peng, S., Li, M., & Jiang, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109598 · Cited by 5
Dharuman, G., Hippe, K., Brace, A., Foreman, S., Hatanpää, V., Sastry, V. K., Zheng, H., Ward, L. T., Muralidharan, S., Vasan, A. K., et al. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00013 · Cited by 9
Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods
Suyunu, B., Taylan, E., & Özgür, A. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822826 · Cited by 6
Phogat, A., Krishnan, S. R., Pandey, M., & Gromiha, M. M. (2025) · Methods · doi:10.1016/j.ymeth.2025.01.020 · Cited by 5
bindNode24: Competitive binding residue prediction with 60 % smaller model
Erckert, K., Birkeneder, F., & Rost, B. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.02.042 · Cited by 5
“The F1 performance for the other four (ProstT5 [13], ProtT5 [27], ESM-2/3b [29] and Ankh [28]) did not differ within the 95 % confidence interval (CI), i.e., differed by fewer than ± 1.96 standard errors (other performance measures behaved alike, Table S10).”
InfluentialSST-ResNet: A Sequence and Structure Information Integration Model for Protein Property Prediction
Zhou, G., Zhao, Y., He, S., & Bo, X. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26062783 · Cited by 5
“The following supporting information can be downloaded at: https:// www.mdpi.com/article/10.3390/ijms26062783/s1.”
InfluentialAn integrated AI knowledge graph framework of bacterial enzymology and metabolism
Spencer, N. R., Gunabalasingam, M., Dial, K., Di, X., Malcolm, T., & Magarvey, N. A. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2425048122 · Cited by 5
AI-Empowered Genome Decoding: Applications of Large Language Models in Genomics
Li, S., Fan, W., & Zhou, Y. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0051-1 · Cited by 4
A fast (CNN + MCWS-transformer) based architecture for protein function prediction
Mahala, A., Ranjan, A., Priyadarshini, R., Vikram, R., & Dansena, P. (2025) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/sagmb-2024-0027 · Cited by 5
Zhang, Y., Qin, Y., Pourmirzaei, M., Shao, Q., Wang, D., & Xu, D. H. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_2 · Cited by 5
Ko, Y. S., Parkinson, J., & Wang, W. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag014 · Cited by 1
“We exclude the disorder [44] and secondary structure prediction [45, 46] tasks because protein structure tasks have already been shown to be well aligned with current pLM pre-training [18].”
Data-driven design of orthogonal protein-protein interactions
Malinverni, D., & Babu, M. M. (2023) · Science Signaling · doi:10.1126/scisignal.abm4484 · Cited by 7
Yu, X., Negron, C., Huang, L., & Veldman, G. M. (2023) · Antibody Therapeutics · doi:10.1093/abt/tbad011 · Cited by 5
Hasani, M., Trost, C. N., Timmerman, N. J., & Jin, L. (2023) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3584371.3613007 · Cited by 5
Li, X., Wang, G.-A., Wei, Z., Wang, H., & Zhu, X. (2023) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2023.107970 · Cited by 5
Determining epitope specificity of T-cell receptors with transformers
Khan, A. R., Reinders, M. J. T., & Khatri, I. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad632 · Cited by 5
Evaluation of input data modality choices on functional gene embeddings
Brechtmann, F., Bechtler, T., Londhe, S., Mertes, C., & Gagneur, J. (2023) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqad095 · Cited by 6
Lee, J. S., Karthikeyan, D., Fini, M., Vincent, B. G., & Rubinsteyn, A. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad495 · Cited by 5
“Recent advances in protein language models such as ProtBERT [33], ProtT5 [33] and ESM [34] have demonstrated a powerful capacity to capture diverse functional features from protein sequence alone.”
BackgroundPogány, D., & Antal, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300906 · Cited by 6
“We discarded duplicated compounds lacking a SMILES descriptor or containing more than 100 non-H atoms, as well as compounds for which we could not produce an input representation.”
An evolutionary theory on virus mutation in COVID-19
Luo, L., & Lv, J. (2024) · Virus Research · doi:10.1016/j.virusres.2024.199358 · Cited by 5
TabFedSL: A Self-Supervised Approach to Labeling Tabular Data in Federated Learning Environments
Wang, R., Hu, Y., Chen, Z., Guo, J., & Liu, G. (2024) · Mathematics · doi:10.3390/math12081158 · Cited by 5
Foundation models in molecular biology
Si, Y., Zou, J., Gao, Y., Chuai, G., Liu, Q., & Chen, L. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240006 · Cited by 6
Gao, M., Song, C., & Liu, T. (2024) · Journal of Cellular Biochemistry · doi:10.1002/jcb.30642 · Cited by 5
“In this study, we utilized seven pre ‐ trained PLMs to extract the embeddings from the protein sequence, including ProtT5 [26], ProtBert [26], ProtAlbert [26], ProtXLNet [26], TAPE [25], ESM2 [24], and ESM ‐ 1b [36].”
Unsupervised Graph Anomaly Detection on Directed Attribute Network
Li, H., Wang, S., Liu, X., & Gan, X. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651283 · Cited by 5
ProBID-Net: a deep learning model for protein–protein binding interface design
Chen, Z., Ji, M., Qian, J., Zhang, Z., Zhang, X., Gao, H., Wang, H., Wang, R., & Qi, Y. (2024) · Chemical Science · doi:10.1039/d4sc02233e · Cited by 8
Nguyen, V. T. D., Nguyen, N. D., & Hy, T. S. (2024) · Structural Dynamics · doi:10.1063/4.0000271 · Cited by 8
Wang, X., Zhang, M., Yang, X., Yu, D., & Ge, F. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01999 · Cited by 5
A comprehensive review of computational methods for Protein-DNA binding site prediction
Liu, Z., Qiu, W., Liu, Y., He, Y., Pei, W., Zhu, Y.-H., & Qiu, J. (2025) · Analytical Biochemistry · doi:10.1016/j.ab.2025.115862 · Cited by 5
“Taking ULDNA [15] as an example, it inherits the sequence feature embeddings from three large language models (i.e., ESM2 [88], ProtTrans [87], and ESM-MSA [92]), which are then fed to an LSTM-attention architecture for implementing a high-accuracy DNA-binding site prediction model.”
Annotating the microbial dark matter with HiFi-NN
Ayres, G., Munsamy, G., Heinzinger, M., Ferruz, N., Yang, K., Bergman, B., & Lorenz, P. (2025) · iScience · doi:10.1016/j.isci.2025.112480 · Cited by 8
Gao, L., Zhang, Y., Ge, F., Li, S., Guo, Y., Song, J., & Yu, D. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00055 · Cited by 5
GOProteinGNN: Leveraging Protein Knowledge Graphs for Protein Representation Learning
Kalifa, D., Singer, U., & Radinsky, K. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761500 · Cited by 8
Decoding disorder: Advances in predicting intrinsically disordered proteins
Chaurasiya, D., & Lahiri, T. (2025) · Current Proteomics · doi:10.1016/j.curpro.2025.100030 · Cited by 5
Interpretable Kolmogorov-Arnold networks for enzyme commission number prediction
Dumontet, L., Han, S., Prouvost, A., Lee, J. H., Oh, T., & Kang, M. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-025-00059-x · Cited by 5
Kotowski, K., Fabian, P., Roterman, I., & Stąpor, K. (2022) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26452 · Cited by 6
“23 This speed advantage has just been demonstrated at a huge scale by the META AI Team.”
BackgroundMethodInfluentialHasan, A. K. M. M., Ahmed, A. Y., Mahbub, S., Rahman, M. S., & Bayzid, M. S. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad042 · Cited by 5
FRAGSITE2 : A structure and fragment‐based approach for virtual ligand screening
Zhou, H., & Skolnick, J. (2023) · Protein Science · doi:10.1002/pro.4869 · Cited by 4
Reliability of the In Silico Prediction Approach to In Vitro Evaluation of Bacterial Toxicity
Ahn, S.-Y., Kim, M., Bae, J.-E., Bang, I. S., & Lee, S.-W. (2022) · Sensors · doi:10.3390/s22176557 · Cited by 5
“As suggested by Elnaggar, it has more computation layers than the original BERT implementation and is pretrained using protein sequences from UniRef and BFD [18].”
MethodAssessing the performance of protein regression modelsPreprint
Richard, M., Kæstel‐Hansen, J., Groth, P. M., Bartels, S., Salomon, J., Tian, P., Hatzakis, N. S., & Boomsma, W. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.18.545472 · Cited by 5
“Advances in representation learning have shown that different approaches to unsupervised learning from biological sequences lead to differently structured representations (8;12;13;24;25;20).”
BackgroundMethodInfluentialProtein Language Model Supervised Precise and Efficient Protein Backbone Design MethodPreprint
Zhang, B., Liu, K., Zheng, Z., Liu, Y., Mu, J., Wei, T., & Chen, H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.26.564121 · Cited by 5
Martinez, Z. A., Murray, R. M., & Thomson, M. W. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.24.563881 · Cited by 7
Hoffbauer, T., & Strodel, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.12.575432 · Cited by 5
A new framework for evaluating model out-of-distribution generalisation for the biochemical domainPreprint
Fernández-Díaz, R., Shields, D. C., Hoang, T. L., & López, V. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.14.584508 · Cited by 6
MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-TrainingPreprint
Chen, B., Bei, Z., Cheng, X., Li, P., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.10.598380 · Cited by 11
Thompson, M., Martín, M., Olmo, T. S., Rajesh, C., Koo, P. K., Bolognesi, B., & Lehner, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.13.603366 · Cited by 5
Hasanaj, E., Cole, E., Mohammadi, S., Addagudi, S., Zhang, X., Song, L., & Xing, E. P. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.26.661186 · Cited by 7
Ghaly, T. M., Rajabal, V., Russell, D., Colombi, E., & Tetu, S. G. (2025) · doi:10.1101/2025.04.02.646905 · Cited by 5
BiRNA-BERT allows efficient RNA language modeling with adaptive tokenization
Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., & Bayzid, M. S. (2025) · Communications Biology · doi:10.1038/s42003-025-08982-0 · Cited by 16
Zhang, P., Ma, J., & Chen, T. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3576488 · Cited by 13
Pseudo-perplexity in one fell swoop for protein fitness estimation
Kantroo, P., Wagner, G. P., & Machta, B. B. (2025) · PRX Life · doi:10.1103/zhx7-hcmm · Cited by 8
A paired sequence language model for protein-protein interaction modeling
Liu, J., Chen, H., & Zhang, Y. (2026) · Nature Communications · doi:10.1038/s41467-026-70457-5 · Cited by 8
GOBoost: leveraging long-tail gene ontology terms for accurate protein function prediction
Zhang, L., Wang, Y., Chen, X., Hou, J., Si, D., Ding, R., Jiang, B., Ledenko, H., & Cao, R. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf267 · Cited by 5
StackDPP: a stacking ensemble based DNA-binding protein prediction model
Ahmed, S. H., Bose, D. B., Khandoker, R., & Rahman, M. S. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05714-9 · Cited by 5
Lu, J., Chen, H., & Qiu, J. (2024) · IEEE Access · doi:10.1109/access.2024.3427792 · Cited by 4
“We conducted a comparison of various protein language models, including Seqvec [48], Esm [50], and Protans [51], ultimately selecting Protans as the embedding feature for HGNet.”
MethodLi, Y., Nan, X., Zhang, S., Zhou, Q., Lu, S., & Tian, Z. (2024) · Biomolecules · doi:10.3390/biom14101220 · Cited by 4
“By leveraging the ProtT5 language model [38] for feature extraction, PMSFF efficiently generates comprehensive protein representations without relying on external tools or time-consuming feature engineering.”
MethodPre-trained Language Models for Decoding Protein Language: a Survey
Amjad, M., Munir, A., Zia, U., & Paracha, R. Z. (2024) · International Conference on Digital Futures and Transformative Technologies (ICoDT2) · doi:10.1109/icodt262145.2024.10740205 · Cited by 4
Thomas, D. P. G., Garcia Fernandez, C. M., Haydarlou, R., & Feenstra, K. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-88445-y · Cited by 4
Chuang, C.-C., Liu, Y.-C., Jhang, W.-E., Wei, S.-S., & Ou, Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02144 · Cited by 4
Jeon, J., Yu, S., Lee, S., Kim, S. C., Jo, H.-Y., Jung, I., & Kim, K. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf080 · Cited by 4
Qi, F., Huang, Q., Yao, X., Cao, Y., Shen, Y., Ren, Y., Liu, Z., & Zhang, Z. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf327 · Cited by 4
“Pretrained protein language models include UniRep [49], ESM [50], ProtTrans [51], Progen [52], ProtGPT2 [53], and xTrimoPGLM [54].”
Metadata
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Jiangnan University (CN)
- Google (United States) (US)
- Nvidia (United States) (US)
- NVIDIA, Santa Clara, CA, USA
- Seoul National University (KR)
- Oak Ridge National Laboratory (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 7112-7127
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3177500196
- Cited by
- 2,329
- References
- 101
- Citations by year
- 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
- New Faculty Startup Fund and the Creative-Pioneering Researchers Program
- Seoul National University
- Oak Ridge National Laboratory
- U.S. Department of Energy (DE-AC05-00OR22725)
- TensorFlow Research Cloud
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
- MeSH terms
- Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
