The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index.
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Open accessPublished version of a preprint
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane versus water-soluble (2-state accuracy Q2=91%). For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that pLMs learned some of the grammar of the language of life. All our models are available through https://github.com/agemagician/ProtTrans.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Fine-tuning has not been extensively explored in previous work5, possibly due to its demanding computational requirements.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [5]
In the citing paper
“Such models were then applied to protein sequence: the BERT architecture was directly trained on the Uniref100 and BFD-100 datasets to create the ProtBERT model [58] for protein sequence prediction.”
From Computational and artificial intelligence-based methods for antibody development (2023) · cites this paper as [58]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
(2008) · Journal of Machine Learning Research · Cited by 33,051
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Research works printed but not linked: 5
- john, 2020, High accuracy protein structure prediction using deep learning, Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book)
- limited, 2019, Press release announcing Supercomputer Fugaku
- rao, 2021, MSA transformer, bioRxiv 2021 02 12 430858
- armenteros, 2020, Language modelling for biological sequences–curated datasets and baselines, bioRxiv 2020 03 09 983585
- 10.1110/ps.03128904 · DOI
Cited by
Sorted by citations
Items 701–800 of 1,007
Cell type matching across species using protein embeddings and transfer learning
Biharie, K., Michielsen, L., Reinders, M. J. T., & Mahfouz, A. E. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad248 · Cited by 9
Liyaqat, T., Ahmad, T., & Saxena, C. (2023) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-023-00533-1 · Cited by 10
“ProtTrans [43] offers cutting-edge pre-trained protein language models.”
BackgroundIntelligent Protein Design and Molecular Characterization Techniques: A Comprehensive Review
Wang, J., Chen, C., Yao, G., Ding, J., Wang, L., & Jiang, H. (2023) · Molecules · doi:10.3390/molecules28237865 · Cited by 10
“…successors GPT-2 and GPT-3 [170–172], the Evolutionary Scale Modeling (ESM) family for predicting protein structure and function (ESM-1b, ESM-MSA-1b, & ESM-1v) [173–175], the ProtTrans with the largest training dataset [176], and the ProGen language model [177] that can control protein generation.”
BackgroundGhazikhani, H., & Butler, G. (2024) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26694 · Cited by 10
Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language ModelPreprint
He, L., Zhang, S., Wu, L., Xia, H., Ju, F., Zhang, H., Liu, S., Xia, Y., Zhu, J., Deng, P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15527 · Cited by 35
“ProtTrans (Elnaggar et al., 2020) trains the protein LMs (pLMs) on the Summit supercomputer using more than 4 thousand GPUs and TPU Pod up-to 1024 cores, the most informative embeddings even outperform state-of-the-art models without multiple sequence alignments (MSAs) for the secondary structure…”
MethodLearning sequence, structure, and function representations of proteins with language modelsPreprint
Hamamsy, T., Barot, M., Morton, J. T., Steinegger, M., Bonneau, R., & Cho, K. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.26.568742 · Cited by 12
Chen, T., Quinn, Z., Mishra, K., O’Connor, E. C., Silver, S. E., Zhang, Y., Valencia, M. J., Mei, Y., Behmoaras, J., Ferreira, L. M. R., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.31.606098 · Cited by 11
Distilling Structural Representations into Protein Sequence ModelsPreprint
Ouyang-Zhang, J., Gong, C., Zhao, Y., Krähenbühl, P., Klivans, A. R., & Diaz, D. J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.08.622579 · Cited by 12
Scoring alignments by embedding vector similarity
(2024) · doi:10.1093/bib/bbae178 · Cited by 8
Johnson, S. R., Weigele, P. R., Fomenkov, A., Ge, A., Vincze, A., Eaglesham, J. B., Roberts, R. J., & Sun, Z. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkae1175 · Cited by 10
Discovery of highly active kynureninases for cancer immunotherapy through protein language model
Eom, H., Park, S., Cho, K. S., Lee, J., Kim, H., Kim, S., Yang, J., Han, Y.-H., Lee, J., Seok, C., et al. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkae1245 · Cited by 10
Exploration of protein sequence embeddings for protein-ligand binding site detection
Hoksza, D., & Gamouh, H. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995025 · Cited by 8
“There exist different techniques for generating protein embeddings such as ProtVec [19], UDSMProt [20], SeqVec [21], ESM-1b [22], UniRep [23], and ProtTrans [24].”
BackgroundMethodZhang, Y., Bai, Z., & Imoto, S. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad617 · Cited by 14
Chomicz, D., Kończak, J., Wróbel, S., Satława, T., Dudzic, P., Janusz, B., Tarkowski, M., Deszyński, P., Gawłowski, T., Kostyn, A., et al. (2024) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2024.1352508 · Cited by 9
Drug–Target Interactions Prediction at Scale: The Komet Algorithm with the LCIdb Dataset
Guichaoua, G., Pinel, P., Hoffmann, B., Azencott, C., & Stoven, V. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00422 · Cited by 11
Qiao, B., Wang, S., Hou, M., Chen, H., Zhou, Z., Xie, X., Pang, S., Yang, C., Yang, F., Zou, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae581 · Cited by 9
PatchProt: hydrophobic patch prediction using protein foundation models
Gogishvili, D., Minois-Genin, E., van Eck, J., & Abeln, S. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae154 · Cited by 11
Alg-MFDL: A multi-feature deep learning framework for allergenic proteins prediction
Hu, X., Li, J., & Liu, T. (2024) · Analytical Biochemistry · doi:10.1016/j.ab.2024.115701 · Cited by 9
IDP-Bert: Predicting Properties of Intrinsically Disordered Proteins Using Large Language Models
Mollaei, P., Sadasivam, D., Guntuboina, C., & Farimani, A. B. (2024) · The Journal of Physical Chemistry B · doi:10.1021/acs.jpcb.4c02507 · Cited by 13
Hyena architecture enables fast and efficient protein language modeling
Zhang, Y., Bian, B., & Okumura, M. (2024) · iMetaOmics. · doi:10.1002/imo2.45 · Cited by 10
Tzavella, K., Díaz, A., Olsen, C., & Vranken, W. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae664 · Cited by 9
“The best-performing pLM for our task was ProtT5-XL [31], trained on the BFD100 [46] and Uniref50 [47] datasets, from which we extracted 1024-dimensional (1024-”
InfluentialPredicting purification process fit of monoclonal antibodies using machine learning
Maier, A. J., Cha, M., Burgess, S., Wang, A., Cuellar, C., Kim, S. Y., Rajan, N. S., Neyyan, J., SENGUPTA, R., O’Connor, K., et al. (2025) · mAbs · doi:10.1080/19420862.2024.2439988 · Cited by 11
Song, J., & Kurgan, L. (2025) · Expert Review of Proteomics · doi:10.1080/14789450.2025.2451715 · Cited by 9
Wang, W., Shuai, Y., Li, Y., Zeng, M., & Li, M. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3529301 · Cited by 9
Lee, K. W., Pham, N. T., Min, H. J., Park, H. W., Lee, J., Lo, H., Kwon, N. Y., Seo, J.-M., Shaginyan, I., Cho, H., et al. (2025) · Journal of Molecular Biology · doi:10.1016/j.jmb.2025.168977 · Cited by 9
Kim, S., Lee, W., Kim, H. I., Kim, M. K., & Choi, T. S. (2025) · Molecules and Cells · doi:10.1016/j.mocell.2025.100191 · Cited by 14
“This approach is trained to learn billions of protein sequence data with the protein language model (Ruffolo and Madani, 2024) based on ProtT5 (Elnaggar et al., 2022) to extract important patterns and information from each protein ’ s sequence.”
Lü, X., Fan, S., Lü, R., Zong, L., Xia, Z., Jin, Y., Zhang, Z., & Yang, Z. (2025) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2025.141024 · Cited by 9
Kallergis, G., Asgari, E., Empting, M., Hirsch, A. K. H., Klawonn, F., & McHardy, A. C. (2025) · Communications Chemistry · doi:10.1038/s42004-025-01484-4 · Cited by 10
Yin, M., Zhou, H., Wu, J., Zhu, Y., Zhan, Y., Kong, Z., Xu, H., Hsieh, C., Chen, J., Hou, T., et al. (2025) · Research · doi:10.34133/research.0721 · Cited by 12
Protein sequence classification using natural language processing techniques
Perveen, H., & Weeds, J. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00304-x · Cited by 9
Xie, P., Yao, L., Guan, J., Chung, C., Zhao, Z., Long, F., Sun, Z., Lee, T.-Y., & Chiang, Y. (2025) · Protein Science · doi:10.1002/pro.70087 · Cited by 10
“Building the classification model without ProtBert-BFD reduces evolutionary features, respectively, providing a solid foundation for extracting high-level hidden features.”
InfluentialCharting γ-secretase substrates by explainable AI
Breimann, S., Kamp, F., Basset, G., Abou‐Ajram, C., Güner, G., Yanagida, K., Okochi, M., Müller, S. A., Lichtenthaler, S. F., Langosch, D., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60638-z · Cited by 9
AllerTrans: a deep learning method for predicting the allergenicity of protein sequences
Sarlakifar, F., Malek, H., Fard, N. A., & Khotanlou, Z. (2025) · Biology Methods and Protocols · doi:10.1093/biomethods/bpaf040 · Cited by 9
“Two pLMs, ESM-2 (11), and ProtT5 [20] were employed to extract embedding vectors from input protein sequences in FASTA format.”
Nguyen, N. D. H., Pham, N. T., Tran, D. T., Wei, L., Malik, A., & Manavalan, B. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-01078-1 · Cited by 11
SPACE: STRING proteins as complementary embeddings
Hu, D., Szklarczyk, D., von Mering, C., & Jensen, L. J. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf496 · Cited by 11
Deep Learning for Drug–Target Interaction Prediction: A Comprehensive Review
Chen, Y., Luo, D., & Xue, W. (2025) · Chemical Biology & Drug Design · doi:10.1111/cbdd.70183 · Cited by 9
Mechanism‐Driven Screening of Membrane‐Targeting and Pore‐Forming Antimicrobial Peptides
Li, J., Yang, C., Dong, R., Juarez, J. F. B., Wang, L., Wettstein, M. E., Wang, D., Cao, C., Lü, Y., & Song, C. (2025) · Advanced Science · doi:10.1002/advs.202516470 · Cited by 9
RP3Net: a deep learning model for predicting recombinant protein production in Escherichia coli
Tankhilevich, E., Cuesta, S. M., Barrett, I. P., Berg, C., Schiavone, L. H., & Leach, A. R. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag003 · Cited by 11
Protein folding stability estimation with explicit consideration of unfolded states
Lee, H. C., Cho, Y., Yun, J., Steinegger, M., Kim, H. M., & Park, H. (2026) · Nature Communications · doi:10.1038/s41467-026-68637-4 · Cited by 11
Wu-Mo, C., Flores-González, A., Meléndez-Delgado, J., Ortiz-Gómez, V., Meléndez-González, H., & Maldonado‐Hernández, R. (2026) · Microorganisms · doi:10.3390/microorganisms14030591 · Cited by 9
CAMPER: mechanistic artificial intelligence for designing peptides that target MRSA persisters
Shehadeh, F., Mishra, B., Ferrer‐Espada, R., Basu, A., Felix, L. O., Dellis, C., Ganesan, N., Zhang, L., Martens, A. T., Goulev, Y., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-70348-9 · Cited by 10
ManyFold: an efficient and flexible library for training and validating protein folding models
Villegas-Morcillo, A., Robinson, L., Flajolet, A., & Barrett, T. D. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac773 · Cited by 10
Anteghini, M., dos Santos, V. A. P. M., & Saccenti, E. (2023) · Journal of Cellular Biochemistry · doi:10.1002/jcb.30490 · Cited by 9
Mapping Data to Deep Understanding: Making the Most of the Deluge of SARS-CoV-2 Genome Sequences
Sokhansanj, B. A., & ROSEN, G. L. (2022) · mSystems · doi:10.1128/msystems.00035-22 · Cited by 9
Ghazikhani, H., & Butler, G. (2022) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb55180.2022.9863026 · Cited by 9
“2) ProtBERT-BFD: We used the ProtBERT-BFD model from ProtTrans [18] which is pre-trained on the BFD database with around 2.5 billion protein sequences, or approximately 400 billion amino acid sequences [21] (https://bfd.mmseqs. com) which integrates UniProt with proteins translated from metagenomic…”
BackgroundMethodInfluentialZhu, G., Fan, Y., Li, F., Choi, A. T. H., Tan, Z., Cheng, Y., Li, K., Wang, S., Luo, C., Liu, H., et al. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120439 · Cited by 10
How to select the best model from AlphaFold2 structures?Preprint
Takei, Y., & Ishida, T. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.05.487218 · Cited by 11
“We used the standard version of DeepAccNet and DeepAccNet-Bert, which adds sequence embedding by ProtTrans[44].”
MethodStability Oracle: A Structure-Based Graph-Transformer for Identifying Stabilizing MutationsPreprint
Diaz, D. J., Gong, C., Ouyang-Zhang, J., Loy, J. M., Wells, J., Yang, D. K., Ellington, A. D., Dimakis, A. G., & Klivans, A. R. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.05.15.540857 · Cited by 9
Pourmirzaei, M., Esmaili, F., Pourmirzaei, M., Wang, D.-L., & Xu, D. (2024) · bioRxiv · doi:10.1101/2024.05.31.596915 · Cited by 12
Allman, B. E., Vieira, L., Diaz, D. J., & Wilke, C. O. (2025) · Journal of The Royal Society Interface · doi:10.1098/rsif.2024.0598 · Cited by 11
Vidal-Henriquez, E., Holder, T., Lee, N. F., Pompe, C., & Teese, M. G. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.10.026 · Cited by 9
“Advanced protein representations were also generated via protein language model embeddings using the ProtTrans prot_t5_xl_half_uniref50 model [42] through the provided prott5_em-bedder.py script.”
Accurate top protein variant discovery via low-N pick-and-validate machine learning
Chu, H. Y., Fong, J. H. C., Thean, D., Zhou, P., Fung, F. K. C., Huang, Y., & Wong, A. S. L. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.002 · Cited by 10
Zhang, M., Wang, X., Xu, S., Ge, F., Paixao, I. C., Song, J., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00739 · Cited by 8
Malik, M. S., Chang, Y., Liu, Y., Le, V. T., & Ou, Y. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00961 · Cited by 8
Malusare, A., Kothandaraman, H., Tamboli, D., Lanman, N. A., & Aggarwal, V. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae117 · Cited by 9
“Early applications of foundation models in bioinformatics can be seen in analyzing protein sequences (Rives et al. 2021, Elnaggar et al. 2022), which were then trained on diverse applications like calculation of protein structure, prediction of mutation effects, and the understanding of…”
BackgroundCytometry masked autoencoder: An accurate and interpretable automated immunophenotyper
Kim, J., Ionita, M., Lee, M., McKeague, M. L., Pattekar, A., Painter, M. M., Wagenaar, J., Truong, V., Norton, D. T., Mathew, D., et al. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2024.101808 · Cited by 13
Paranou, D., Chatzigoulas, A., & Cournia, Z. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae078 · Cited by 8
Pratyush, P., Pokharel, S., Ismail, H. D., Bahmani, S., & KC, D. B. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4196-5_16 · Cited by 8
Creanza, T. M., Alberga, D., Patruno, C., Mangiatordi, G. F., & Ancona, N. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02019 · Cited by 8
Computational methods for binding site prediction on macromolecules
Kozlovskii, I., & Popov, P. A. (2025) · Quarterly Reviews of Biophysics · doi:10.1017/s003358352500006x · Cited by 9
Fine-tuning of conditional Transformers improves in silico enzyme prediction and generation
Nicolini, M., Saitto, E., Franco, R. E. J., Cavalleri, E., Galeano, A., Malchiodi, D., Paccanaro, A., Robinson, P. N., Casiraghi, E., & Valentini, G. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.03.037 · Cited by 8
Yu, X., Wang, Y., Chen, L., Li, W., Tang, Y., & Liu, G. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101317 · Cited by 8
“We generated initial protein features using the pre-training model prot_t5_xl_uniref50 from the ProtBert framework [18].”
Tartici, A., Nayar, G., & Altman, R. B. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf330 · Cited by 9
Jang, Y. E., Kwon, M., Kim, S. G., George, N. P., Hwang, J. S., Basith, S., & Lee, G. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110590 · Cited by 8
AI4Protein: transforming the future of protein design
Wang, D., Tan, Z., Gao, J., Zhang, S., Shen, J., & Lu, Y. (2025) · Science China Life Sciences · doi:10.1007/s11427-024-2906-3 · Cited by 12
Ahmed, Z., Shahzadi, K., Li, R., Jiang, Y., Jin, Y.-T., Arif, M., & Feng, J. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf313 · Cited by 8
“These rich embeddings serve as informative input features for a variety of downstream applications, including protein classification [44].”
SeaMoon: From protein language models to continuous structural heterogeneity
Lombard, V., Timsit, D., Grudinin, S., & Laine, E. (2025) · Structure · doi:10.1016/j.str.2025.06.010 · Cited by 8
A survey of downstream applications of evolutionary scale modeling protein language models
Yang, Q., Yu, J., & Zheng, J. (2025) · Quantitative Biology · doi:10.1002/qub2.70013 · Cited by 9
LassoESM a tailored language model for enhanced lasso peptide property prediction
Mi, X., Barrett, S. E., Mitchell, D. A., & Shukla, D. (2025) · Nature Communications · doi:10.1038/s41467-025-63412-3 · Cited by 7
Li, G., Wang, L., Luo, J., & Cheng, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-29666-z · Cited by 8
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 24
A flaw in using pretrained protein language models in protein–protein interaction inference models
Szymborski, J., & Emad, A. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-025-01176-7 · Cited by 13
Dual-encoder contrastive learning accelerates enzyme discovery
Rocks, J. W., Truong, D. P., Rappoport, D., Maddrell-Mander, S., Martin-Alarcon, D. A., Lee, T. M., Crossan, S., & Goldford, J. E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2520070123 · Cited by 9
An, J., Liu, S., Liu, W., Zhang, H., Hu, J., Zhang, C., Jenis, J., Liu, X., & Li, H. (2026) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2026.105730 · Cited by 8
Structure, Interactions, and Assembly of Membrane-Active Antimicrobial Polypeptides
Lee, T.-H., Charchar, P., Sani, M., Le, D.-H., Le, T. C., Yarovsky, I., Separovic, F., & Aguilar, M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00994 · Cited by 12
Scaling SMILES-Based Chemical Language Models for Therapeutic Peptide Engineering
Feller, A. L., Secor, M., Swanson, S., Wilke, C. O., & Deibler, K. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c00652 · Cited by 9
Chen, B., Jin, J., Liu, H., Yang, Z., Zhu, H., Wang, Y., Lin, J., Wang, S., & Chen, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1289669 · Cited by 9
POOE: predicting oomycete effectors based on a pre-trained large protein language model
Zhao, M., Lei, C., Zhou, K., Huang, Y., Fu, C., Yang, S., & Zhang, Z. (2023) · mSystems · doi:10.1128/msystems.01004-23 · Cited by 9
Detecting anomalous proteins using deep representations
Michael-Pitschaze, T., Cohen, N., Ofer, D., Hoshen, Y., & Linial, M. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae021 · Cited by 11
Yang, Q., Jin, X., Zhou, H., Ying, J., Zou, J., Liao, Y., Lu, X., Ge, S., Yu, H. J., & Min, X. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108067 · Cited by 8
Comprehensive Research on Druggable Proteins: From PSSM to Pre-Trained Language Models
Chu, H., & Liu, T. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25084507 · Cited by 8
Improvements in viral gene annotation using large language models and soft alignments
Harrigan, W., Ferrell, B. D., Wommack, K. E., Polson, S. W., Schreiber, Z. D., & Belcaid, M. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05779-6 · Cited by 9
Li, Y., Gao, R., Liu, S., Zhang, H., Lv, H., & Lai, H. (2024) · Methods · doi:10.1016/j.ymeth.2024.08.004 · Cited by 8
Jia, R., He, Z., Wang, C., Guo, X., & Li, F. (2024) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2024.3467093 · Cited by 8
Wang, X., Zhang, Z., & Liu, C. (2024) · Journal of Molecular Biology · doi:10.1016/j.jmb.2024.168810 · Cited by 8
Machine learning approaches for predicting protein-ligand binding sites from sequence data
Vural, O., & Jololian, L. (2025) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2025.1520382 · Cited by 8
Dampier, W., Link, R. W., Earl, J. P., Collins, M. L., Souza, D. R. D., Koser, K. L., Nonnemacher, M. R., & Wigdahl, B. L. (2022) · Frontiers in Virology · doi:10.3389/fviro.2022.880618 · Cited by 8
“The Rost-Lab showed that this model could be further refined in a process called “transfer learning” to predict new tasks such as subcellular localization, secondary structure, and enzyme activity (46).”
BackgroundMethodInfluentialG– PLIP : Knowledge graph neural network for structure-free protein–ligand bioactivity prediction
Crouzet, S., Lieberherr, A. M., Atz, K., Nilsson, T., Sach-Peltason, L., Müller, A. T., Peraro, M. D., & Zhang, J. D. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.06.029 · Cited by 9
The substrate scopes of enzymes: a general prediction model based on machine and deep learningPreprint
Kroll, A., Ranjan, S., Engqvist, M. K. M., & Lercher, M. J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.05.24.493213 · Cited by 8
“In order to create a single fixed-length numerical representation of the whole protein, one typically calculates the element-wise mean across all amino acid representations 33,2,39 .”
BackgroundEfficient and accurate sequence generation with small-scale protein language modelsPreprint
Serrano, Y., Rodà, S., Guallar, V., & Molina, A. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.04.551626 · Cited by 8
Rapid protein evolution by few-shot learning with a protein language modelPreprint
Jiang, K.-Y., Yan, Z.-Q., Bernardo, M. D., Sgrizzi, S. R., Villiger, L., Kayabolen, A., Kim, B., Carscadden, J. K., Hiraizumi, M., Nishimasu, H., et al. (2024) · bioRxiv · doi:10.1101/2024.07.17.604015 · Cited by 9
Advancements in Rational Multi‐Targeted Drug Discovery
Ravikumar, B., Cichońska, A., Sahni, N., Aittokallio, T. A., & Rahman, R. (2024) · Wiley eBooks · doi:10.1002/9781394182862.ch9 · Cited by 8
Which pLM to choose?Preprint
Senoner, T., Koludarov, I., Guenther, J., Shehu, A., Rost, B., & Bromberg, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.30.685515 · Cited by 7
Scoring alignments by embedding vector similarity
Ashrafzadeh, S., Golding, G. B., Ilie, S., & Ilie, L. (2023) · doi:10.1101/2023.08.30.555602 · Cited by 8
“Many protein embeddings have been proposed, including ProtVec [4], SeqVec [10], PRoBERTa [22], MSA-transformer [27], ESM-1b and ESM2 [28], and the models in the ProtTrans project: ProtTXL, ProtBert, ProtXLNet, ProtAlbert, ProtElectra, and ProtT5 [8].”
MethodMULAN: multimodal protein language model for sequence and structure encoding
Frolova, D., Pak, M., Litvin, A., Sharov, I., Ivankov, D., & Oseledets, I. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf117 · Cited by 12
Unravelling the Functional Diversity of Type III Polyketide Synthases in Fungi
Sokolova, N., Denisov, S. S., Hackl, T., & Haslinger, K. (2025) · Angewandte Chemie International Edition · doi:10.1002/anie.202514786 · Cited by 9
Protein language models uncover carbohydrate-active enzyme function in metagenomics
Thurimella, K., Mohamed, A. M. T., Li, C., Vatanen, T., Graham, D. B., Owens, R. M., La Rosa, S. L., Plichta, D. R., Bacallado, S., & Xavier, R. J. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06286-y · Cited by 9
“4) [37, 42] to generate embeddings for each of these sequences using a V100 GPU.”
SAFPred: synteny-aware gene function prediction for bacteria using protein embeddings
Urhan, A., Cosma, B.-M., Earl, A. M., Manson, A. L., & Abeel, T. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae328 · Cited by 8
Changiarath, A., Arya, A., Xenidis, V. A., Padeken, J., & Stelzl, L. S. (2024) · Faraday Discussions · doi:10.1039/d4fd00099d · Cited by 7
Annotation-free prediction of microbial dioxygen utilization
Flamholz, A. I., Goldford, J. E., Richter, P. A., Larsson, E. M., Jinich, A., Fischer, W. W., & Newman, D. K. (2024) · mSystems · doi:10.1128/msystems.00763-24 · Cited by 8
Metadata
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Jiangnan University (CN)
- Google (United States) (US)
- Nvidia (United States) (US)
- NVIDIA, Santa Clara, CA, USA
- Seoul National University (KR)
- Oak Ridge National Laboratory (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 7112-7127
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3177500196
- Cited by
- 2,329
- References
- 101
- Citations by year
- 2026: 664 · 2025: 750 · 2024: 533 · 2023: 257 · 2022: 100
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM (Technical University of Munich) (submittedVersion, other-oa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation of Korea (2019R1A6A1A10073437, NRF-2020M3A9G7103933)
- New Faculty Startup Fund and the Creative-Pioneering Researchers Program
- Seoul National University
- Oak Ridge National Laboratory
- U.S. Department of Energy (DE-AC05-00OR22725)
- TensorFlow Research Cloud
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- protein language models, self-supervised learning, protein embeddings, secondary structure prediction, protein localization, ProtT5, Transformer-XL, XLNet, BERT, BFD, dimensionality reduction, protein sequence representation
- MeSH terms
- Supervised Machine Learning, Algorithms, Natural Language Processing, Proteins, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
