Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models
Open accessAbstract
Large pretrained protein language models (PLMs) have improved protein property and structure prediction from sequences via transfer learning, in which weights and representations from PLMs are repurposed for downstream tasks. Although PLMs have shown great promise, currently there is little understanding of how the features learned by pretraining relate to and are useful for downstream tasks. We perform a systematic analysis of transfer learning using PLMs, conducting 370 experiments across a comprehensive suite of factors including different downstream tasks, architectures, model sizes, model depths, and pretraining time. We observe that while almost all down-stream tasks do benefit from pretrained models compared to naive sequence representations, for the majority of tasks performance does not scale with pretraining, and instead relies on low-level features learned early in pretraining. Our results point to a mismatch between current PLM pretraining paradigms and most applications of these models, indicating a need for better pretraining methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous work shows that improvement beyond this performance range is difficult for protein language models with self-supervised pretraining alone.
“Previous work has shown that improvement beyond this performance range is difficult for protein language models with self-supervised pretraining alone (49)”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [49]
The Results (section 5; page unavailable) say that performance on many downstream tasks will not improve as protein language models improve. The Discussion (section 6; page unavailable) similarly says that simply scaling existing pretraining strategies will not improve performance for many tasks. The paper does not establish a general ceiling for all tasks or identify the particular performance range referenced in the citing sentence. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698 · see citing works
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,865 · see citing works
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 14× · see citing works
“For concision, we refer to CARP-38M and ESM-43M as the “small” models ( ), CARP-76M and ESM-85M as the “medium” models ( ), and CARP-640M and ESM-650M (ESM-1b) as the “large” models ( ).”
BackgroundMethodInfluentialSignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01156-3 · Cited by 2,999 · see citing works
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036 · see citing works
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928 · see citing works
SignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Armenteros, J. J. A., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von HEIJNE, G., & Nielsen, H. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000524414 · Cited by 1,905 · see citing works
Exploring protein fitness landscapes by directed evolution
Romero, P. A., & Arnold, F. H. (2009) · Nature Reviews Molecular Cell Biology · doi:10.1038/nrm2805 · Cited by 1,163 · see citing works
DeepLoc: prediction of protein subcellular localization using deep learning
Armenteros, J. J. A., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx431 · Cited by 1,144 · see citing works
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · cited 2× · see citing works
“…pretrained PLMs have been proposed in the past few years (Rives et al., 2021; Yang et al., 2022; El-naggar et al., 2022; Brandes et al., 2022; Alley et al., 2019; Rao et al., 2019; Elnaggar et al., 2023; Lin et al., 2023), these works primarily focus on validating that pretraining improves…”
BackgroundProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · see citing works
DeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 723 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635 · see citing works
Natural Selection and the Concept of a Protein Space
Smith, J. (1970) · Nature · doi:10.1038/225563a0 · Cited by 488 · see citing works
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455 · see citing works
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455 · see citing works
High-resolution <i>de novo</i> structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392 · see citing works
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338 · see citing works
Critical assessment of methods of protein structure prediction (CASP)—Round XII
Moult, J., Fidelis, K. A., Kryshtafovych, A., Schwede, T. F., & Tramontano, A. L. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25415 · Cited by 327 · see citing works
Deep diversification of an AAV capsid protein by machine learning
Bryant, D., Bashir, A., Sinai, S., Jain, N. K., Ogden, P. J., Riley, P. F., Church, G. M., Colwell, L. J., & Kelsic, E. D. (2021) · Nature Biotechnology · doi:10.1038/s41587-020-00793-4 · Cited by 302 · see citing works
Adaptation in protein fitness landscapes is facilitated by indirect paths
Wu, N. C., Dai, L., Olson, C. A., Lloyd‐Smith, J. O., & Sun, R. X. (2016) · eLife · doi:10.7554/elife.16965 · Cited by 281 · see citing works
Meltome atlas—thermal proteome stability across the tree of life
Jarząb, A., Kurzawa, N., Hopf, T. A., Moerch, M., Zecha, J., Leijten, N. M., Bian, Y., Musiol, E., Maschberger, M., Stoehr, G., et al. (2020) · Nature Methods · doi:10.1038/s41592-020-0801-4 · Cited by 275 · see citing works
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 251 · see citing works
Sixty-five years of the long march in protein secondary structure prediction: the final stretch?
Yang, Y., Gao, J., Wang, J., Heffernan, R., Hanson, J., Paliwal, K. K., & Zhou, Y. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw129 · Cited by 237 · see citing works
Informed training set design enables efficient machine learning-assisted directed protein evolution
Wittmann, B. J., Yue, Y., & Arnold, F. H. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.008 · Cited by 208 · see citing works
Self-citationAnkh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 201 · see citing works
Learning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 163 · see citing works
xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of ProteinPreprint
Chen, B., Cheng, X., Li, P., Geng, Y.-A., Gong, J., Li, S., Bei, Z.-L., Tan, X., Wang, B., Zeng, X., et al. (2024) · bioRxiv · doi:10.1101/2023.07.05.547496 · Cited by 157 · see citing works
Protein language models learn evolutionary statistics of interacting sequence motifs
Zhang, Z., Wayment-Steele, H. K., Brixi, G., Wang, H., Kern, D., & Ovchinnikov, S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2406285121 · Cited by 140 · see citing works
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137 · see citing works
Self-citationLight attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130 · see citing works
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 127 · see citing works
Self-citationMachine learning modeling of family wide enzyme-substrate specificity screens
Goldman, S., Das, R., Yang, K. K., & Coley, C. W. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009853 · Cited by 120 · see citing works
Self-citationWhat Makes Transfer Learning Work for Medical Images: Feature Reuse & Other Factors
Matsoukas, C., Haslum, J. F., Sorkhei, M., Söderberg, M. P., & Smith, K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00901 · Cited by 113 · see citing works
Masked inverse folding with sequence transfer for protein representation learning
(2022) · doi:10.1093/protein/gzad015 · Cited by 110 · see citing works
Self-citationLarge language models improve annotation of prokaryotic viral proteins
Flamholz, Z. N., Biller, S. J., & Kelly, L. (2024) · Nature Microbiology · doi:10.1038/s41564-023-01584-8 · Cited by 92 · see citing works
Protein Engineering: Past, Present, and Future
Lutz, S., & Iamurri, S. (2017) · Methods in molecular biology · doi:10.1007/978-1-4939-7366-8_1 · Cited by 91 · see citing works
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 84 · see citing works
Cherti, M., & Jitsev, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892393 · Cited by 26 · see citing works
Combining evolutionary and assay-labelled data for protein fitness predictionPreprint
Hsu, C., Nisonoff, H. M., Fannjiang, C., & Listgarten, J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.03.28.437402 · Cited by 24 · see citing works
Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language ModelPreprint
He, L., Zhang, S., Wu, L., Xia, H., Ju, F., Zhang, H., Liu, S., Xia, Y., Zhu, J., Deng, P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15527 · Cited by 20 · see citing works
Language modelling for biological sequences – curated datasets and baselinesPreprint
Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.09.983585 · Cited by 18 · see citing works
Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?Preprint
Grigg, T. G., Busbridge, D., Ramapuram, J., & Webb, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00528 · Cited by 14 · see citing works
Mansoor, S., Baek, M., Madan, U., & Horvitz, E. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.01.458592 · Cited by 12 · see citing works
Retrieved Sequence Augmentation for Protein Representation LearningPreprint
Ma, C., Zhao, H., Zheng, L., Xin, J., Li, Q., Wu, L., Deng, Z., Yang, L., Liu, Q., & Kong, L. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.02.22.529597 · Cited by 6 · see citing works
Yang, T., Li, M., Tan, P., Zhou, Z., Yu, H., Fan, G., & Hong, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17415 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training. ICLR, 2022.
- Buitinck, L., Louppe, G., Blondel, M., Pedregosa, F., Mueller, A., Grisel, O., Niculae, V., Prettenhofer, P., Gramfort, A., Grobler, J., Layton, R., VanderPlas, J., Joly, A., Holt, B., and Varoquaux, G. API design for machine learning software: experiences from the scikit-learn project. CoRR, abs/1309.0238, 2013. URL http://arxiv.org/abs/1309.0238. · arXiv
- Cuff, J. A. and Barton, G. J. Evaluation and im- provement of multiple sequence methods for pro- tein secondary structure prediction. Proteins, 1999. doi: 10.1002/(SICI)1097-0134(19990301)34:4⟨508:: AID-PROT10⟩3.0.CO;2-4. · DOI
- Hu, M., Yuan, F., Yang, K. K., Ju, F., Su, J., Wang, H., Yang, F., and Ding, Q. Exploring evolution-aware & -free protein language models as protein function predic- tors. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=U8k0QaBgXS.
- Notin, P., Dias, M., Frazer, J., Hurtado, J. M., Gomez, A. N., Marks, D., and Gal, Y. Tranception: protein fitness predic- tion with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learn- ing, pp. 16990-17017. PMLR, 2022.
Cited by
Sorted by citations
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
Genome modelling and design across all domains of life with Evo 2
Brixi, G., Durrant, M. G., Ku, J., Naghipourfar, M., Poli, M., Sun, G., Brockman, G., Chang, D., Fanton, A., Gonzalez, G. A., et al. (2026) · Nature · doi:10.1038/s41586-026-10176-5 · Cited by 431 · see citing works
Self-citationTransformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 148 · see citing works
Genomic language models: opportunities and challenges
Benegas, G., Ye, C., Albors, C., Li, J. C., & Song, Y. S. (2025) · Trends in Genetics · doi:10.1016/j.tig.2024.11.013 · Cited by 133 · see citing works
Zero-shot evaluation reveals limitations of single-cell foundation models
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025) · Genome Biology · doi:10.1186/s13059-025-03574-x · Cited by 109 · see citing works
“Indeed, recent research in protein language models has exposed numerous trivial mechanisms in which transfer learning can appear to boost performance on downstream tasks, but does not actually rely upon sophisticated learning from pretraining [17].”
Self-citationDemocratizing protein language models with parameter-efficient fine-tuning
Sledzieski, S., Kshirsagar, M., Baek, M., Dodhia, R., Lavista Ferres, J., & Berger, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2405840121 · Cited by 86 · see citing works
Evaluating the representational power of pre-trained DNA language models for regulatory genomics
Tang, Z., Somia, N., Yu, Y., & Koo, P. K. (2025) · Genome biology · doi:10.1186/s13059-025-03674-8 · Cited by 63 · see citing works
Addressing epistasis in the design of protein function
Lipsh‐Sokolik, R., & Fleishman, S. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314999121 · Cited by 54 · see citing works
A protein language model for exploring viral fitness landscapes
Ito, J., Strange, A. P., Liu, W., Joas, G., Lytras, S., Matsuno, K., Nao, N., Sawa, H., Mizuma, K., Kojima, I., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-59422-w · Cited by 46 · see citing works
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41 · see citing works
Massive experimental quantification allows interpretable deep learning of protein aggregation
Thompson, M., Martín, M., Olmo, T. S., Rajesh, C., Koo, P. K., Bolognesi, B., & Lehner, B. (2025) · Science Advances · doi:10.1126/sciadv.adt5111 · Cited by 37 · see citing works
AutoPeptideML: a study on how to build more trustworthy peptide bioactivity predictors
Fernández-Díaz, R., Cossio-Pérez, R., Agoni, C., Lam, H. T., López, V., & Shields, D. C. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae555 · Cited by 35 · see citing works
VISH-Pred: an ensemble of fine-tuned ESM models for protein toxicity prediction
Mall, R., Singh, A., Patel, C., Guirimand, G. G. Y., & Castiglione, F. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae270 · Cited by 30 · see citing works
Contrastive learning of T cell receptor representations
Nagano, Y., Pyo, A. G. T., Milighetti, M., Henderson, J., Shawe-Taylor, J. S., Chain, B., & Tiffeau-Mayer, A. (2025) · Cell Systems · doi:10.1016/j.cels.2024.12.006 · Cited by 30 · see citing works
Comprehensive benchmarking of large language models for RNA secondary structure prediction
Zablocki, L. I., Bugnon, L. A., Gerard, M. F., Persia, L. E. D., Stegmayer, G. S., & Milone, D. H. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf137 · Cited by 26 · see citing works
Adams, E., Bai, L., Lee, M., Yu, Y., & AlQuraishi, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.06.636901 · Cited by 23 · see citing works
Benchmarking protein language models for protein crystallization
Mall, R., Kaushik, R., Martinez, Z. A., Thomson, M. W., & Castiglione, F. (2025) · Scientific Reports · doi:10.1038/s41598-025-86519-5 · Cited by 20 · see citing works
Semantical and geometrical protein encoding toward enhanced bioactivity and thermostability
Tan, Y., Zhou, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033.4 · Cited by 20 · see citing works
Evaluation of Machine Learning-Assisted Directed Evolution Across Diverse Combinatorial Landscapes
Li, F.-Z., Yang, J., Johnston, K. E., Gürsoy, E., Yue, Y., & Arnold, F. H. (2024) · Cell Systems · doi:10.1016/j.cels.2025.101387 · Cited by 18 · see citing works
Self-citationAggregating residue-level protein language model embeddings with optimal transport
Naderializadeh, N., & Singh, R. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf060 · Cited by 17 · see citing works
Foundation models of protein sequences: A brief overview
Bjerregaard, A., Groth, P. M., Hauberg, S., Krogh, A., & Boomsma, W. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103004 · Cited by 16 · see citing works
PLMFit: benchmarking transfer learning with protein language models for protein engineering
Bikias, T., Stamkopoulos, E., & Reddy, S. T. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf381 · Cited by 16 · see citing works
Programmable design of functional proteins from natural language
Dai, F., You, S., Zhu, Y., Gao, Y., Fu, L., Zhou, X., Su, J., Wang, C., Fan, Y., Tan, Q., et al. (2026) · Nature · doi:10.1038/s41586-026-11205-z · Cited by 16 · see citing works
Tokenized and continuous embedding compressions of protein sequence and structure
Lu, A. X., Yan, W., Yang, K. K., Gligorijević, V., Cho, K., Abbeel, P., Bonneau, R., & Frey, N. C. (2025) · Patterns · doi:10.1016/j.patter.2025.101289 · Cited by 14 · see citing works
Self-citationDiverse Genomic Embedding Benchmark for functional evaluation across the tree of lifePreprint
West-Roberts, J. A., Kravitz, J., Jha, N., Cornman, A., & Hwang, Y. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.10.602933 · Cited by 12 · see citing works
Deep learning methods for protein representation and function prediction: A comprehensive overview
Wang, M., Nie, Z., He, Y., Vasilakos, A. V., Cheng, Q. S., & Ren, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110977 · Cited by 11 · see citing works
Protein folding stability estimation with explicit consideration of unfolded states
Lee, H. C., Cho, Y., Yun, J., Steinegger, M., Kim, H. M., & Park, H. (2026) · Nature Communications · doi:10.1038/s41467-026-68637-4 · Cited by 11 · see citing works
RP3Net: a deep learning model for predicting recombinant protein production in Escherichia coli
Tankhilevich, E., Cuesta, S. M., Barrett, I. P., Berg, C., Schiavone, L. H., & Leach, A. R. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag003 · Cited by 10 · see citing works
Towards the explainability of protein language models
Hunklinger, A., & Ferruz, N. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01232-w · Cited by 9 · see citing works
Glaser, M., & Brägelmann, J. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.03.635741 · Cited by 8 · see citing works
LassoESM a tailored language model for enhanced lasso peptide property prediction
Mi, X., Barrett, S. E., Mitchell, D. A., & Shukla, D. (2025) · Nature Communications · doi:10.1038/s41467-025-63412-3 · Cited by 7 · see citing works
FLIP2: Expanding Protein Fitness Landscape Benchmarks for Real-World Machine Learning ApplicationsPreprint
Didi, K., Alamdari, S., Lu, A. X., Wittmann, B. J., Johnston, K. E., Amini, A. P., Madani, A., Czeneszew, M., Dallago, C., & Yang, K. K. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.23.707496 · Cited by 7 · see citing works
Self-citationKo, Y. S., Parkinson, J., & Wang, W. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag014 · Cited by 1 · see citing works
FLIGHTED: Inferring fitness landscapes from noisy high-throughput experimental data
Sundar, V., Tu, B., Guan, L., & Esvelt, K. M. (2026) · Cell Systems · doi:10.1016/j.cels.2026.101739 · Cited by 5 · see citing works
Hallee, L., Kapur, R., Patel, A., Gleghorn, J. P., & Khomtchouk, B. B. (2025) · Scientific Reports · doi:10.1038/s41598-025-98185-8 · Cited by 4 · see citing works
Yoshida, K., Hisada, S., Takase, R., Okuma, A., Ishida, Y., Kawara, T., Miura-Yamashita, T., & Ito, D. (2026) · BMC Bioinformatics · doi:10.1186/s12859-026-06401-7 · Cited by 3 · see citing works
Multiobjective learning and design of bacteriophage specificity
Novy, N., Huss, P., Evert, S., Romero, P. A., & Raman, S. (2026) · Cell Systems · doi:10.1016/j.cels.2026.101712 · Cited by 3 · see citing works
Annotation Vocabulary (Might Be) All You NeedPreprint
Hallee, L., Rafailidis, N., Horger, C., Hong, D. K., & Gleghorn, J. P. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.30.605924 · Cited by 3 · see citing works
Semantical and geometrical protein encoding toward enhanced bioactivity and thermostability
Tan, Y., Zhou, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033 · Cited by 2 · see citing works
AbTune: layer-wise selective fine-tuning of protein language models for antibodies
Xu, X., & Bonvin, A. M. J. J. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag374 · Cited by 2 · see citing works
Vinod, R., Amini, A. P., Crawford, L. A., & Yang, K. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013925 · Cited by 2 · see citing works
Self-citationA Survey of Pretrained Protein Language Models
Pokharel, S., Pratyush, P., Chaudhari, M., Heinzinger, M., Caragea, D., Saigo, H., & KC, D. B. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_1 · Cited by 1 · see citing works
Evolutionary Reasoning Does Not Arise in Standard Usage of Protein Language Models
Ektefaie, Y., Shen, A., Jain, L., Farhat, M., & Žitnik, M. (2025) · neural information processing systems · doi:10.52202/085713-4356 · Cited by 1 · see citing works
Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and ThermostabilityPreprint
Tan, Y., Zhong, B., Zheng, L., Fan, G., & Hong, L. (2024) · eLife · doi:10.7554/elife.98033.2 · Cited by 1 · see citing works
Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and ThermostabilityPreprint
Tan, Y., Zhong, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033.3
Predicting protein variant properties with electrostatic representationsPreprint
van der Flier, F. J., van Dijk, A. D. J., de Ridder, D., & Redestig, H. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.11.669649
PEPE: scalable extraction of multi-modal protein language model representations
Zhong, J., Cardente, N., Sandve, G. K., Bashour, H., Abbate, M. F., & Greiff, V. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag375
HyperBind2: Multi-Shot Learning Enables Progressive Improvement in Computational Antibody DiscoveryPreprint
Dell'uomo, D., Satz, A., & Averso, B. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.06.687005
Sequence-Only Prediction of Antibody Fab Thermostability Using Protein Language Model EmbeddingsPreprint
Sennett, M. A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.30.697053
PRIMED: predicting DNA binding residues by leveraging pre-trained protein language models
Zhang, L., Li, X., Song, R., Song, Q., & Fan, X. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1763313
Liang, Y.-X., Wang, Y., Pan, W.-Y., Chen, Z., Wei, J., & Gao, G. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.13.715198
Architectural good practices for reproducible benchmarking in protein machine learning
Vinuesa, J. A. G. , Villegas, D. F., Soto-García, M., Cadet, X. F., Davari, M. D., Cadet, F., Asenjo, J. A., Uribe-Paredes, R., & Medina-Ortiz, D. (2026) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2026.1925740
Barghout, R. A., Wu, S., Chau, W., Barraza-Chavez, J. M., Driza, P., Mahadevan, R., & Sánchez-Lengeling, B. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.10.21.760763
- Authors
- Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K., & Lu, A. X.
- Affiliations
- California Institute of Technology (US)
- Microsoft (United States) (US)
- Corresponding author
- Li, Francesca-Zhoufan
- Lu, Alex X.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2024-02-08
- OpenAlex
- W4391652655
- Cited by
- 51
- References
- 50
- Citations by year
- 2026: 16 · 2025: 22 · 2024: 13
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Funders
- U.S. Department of Energy (DESC0022218)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- protein language models, transfer learning, feature reuse, pre-training, protein property prediction, protein structure prediction, model depth, model size, sequence representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
