CoChat
Sign Up
PreprintCited by 229Machine Learning in BioinformaticsProtein Structure and Dynamicsvaccines and immunoinformatics approaches

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Fine-tuning protein language models boosts predictions across diverse tasks

Open access
Schmirler, Robert, Heinzinger, Michael, Rost, Burkhard·2023·Nature Communications
DOI 10.1101/2023.12.13.571462PMC PMC11358375

Preprint — published as 10.1038/s41467-024-51844-2 · Paper page

Open this paper in CoChat — free account

Abstract

Prediction methods inputting embeddings from protein Language Models (pLMs) have reached or even surpassed state-of-the-art (SOTA) performance on many protein prediction tasks. In natural language processing (NLP) fine-tuning large Language Models (LLMs) has become the de facto standard. In contrast, most pLM-based protein predictions do not back-propagate to the pLM. Here, we compared the fine-tuning of three SOTA pLMs (ESM2, ProtT5, Ankh) on eight different tasks. Two results stood out. Firstly, task-specific supervised fine-tuning almost always improved downstream predictions. Secondly, parameter-efficient fine-tuning could reach similar improvements consuming substantially fewer resources at up to 4.5-fold acceleration of training over fine-tuning full models. Our results suggested to always try fine-tuning, in particular for problems with small datasets, such as for fitness landscape predictions of a single protein. For ease of adaptability, we provided easy-to-use notebooks to fine-tune all models used during this work for per-protein (pooling) and per-residue prediction tasks at https://github.com/RSchmirler/data-repo_plm-finetune-eval .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.12.13.571462

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 73

  1. Vaswani, A. et al. Attention is all you need. Adv. neural information processing systems 30 (2017).
  2. OpenAI. GPT-4 Technical Report. Preprint at https://arxiv.org/abs/2303.08774 (2023). · arXiv
  3. Anil, R. et al. PaLM 2 Technical Report. Preprint at https://arxiv.org/abs/2305.10403 (2023). · arXiv
  4. Bubeck, S. et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4. Preprint at https://arxiv.org/abs/2303.12712 (2023). · arXiv
  5. Liu, Z. et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proc. IEEE/CVF international conference on computer vision 10012–10022 (2021). · DOI · Paper page
  6. 2021, Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, Proc. AAAI conference on artificial intelligence, 35, 11106, 10.1609/aaai.v35i12.17325 · DOI · Paper page
  7. 10.1186/s12859-019-2669-9 · DOI
  8. 2019, Unified rational protein engineering with sequence-based deep representation learning, Nat. methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
  9. Bepler, T. & Berger, B. Learning protein sequence embeddings using information from structure. Preprint at https://arxiv.org/abs/1902.08661 (2019). · arXiv
  10. 10.1073/pnas.2016239118 · DOI · Paper page
  11. Trotter, M. V. , Nguyen, C. Q. , Young, S. , Woodruff, R. T. & Branson, K. M. Epigenomic language models powered by Cerebras. Preprint at https://arxiv.org/abs/2112.07571 (2021). · arXiv
  12. Dalla-Torre, H. et al. The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics. Preprint at https://www.biorxiv.org/content/10.1101/2023.01.11.523679v3 (2023). · DOI
  13. 10.1038/s42256-022-00534-z · DOI · Paper page
  14. 10.1109/TPAMI.2021.3095381 · DOI · Paper page
  15. Elnaggar, A. et al. Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling. Preprint at https://arxiv.org/abs/2301.06568 (2023). · DOI
  16. 10.1126/science.ade2574 · DOI · Paper page
  17. Chen, B. et al. xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein. Preprint at https://www.biorxiv.org/content/10.1101/2023.07.05.547496v3 (2023). · DOI
  18. 10.1186/s12859-022-04873-x · DOI
  19. Ilzhöfer, D. , Heinzinger, M. & Rost, B. SETH predicts nuances of residue disorder from protein embeddings. Front. Bioinforma. 2 (2022). · DOI
  20. 2022, Protein language-model embeddings for fast, accurate, and alignment-free protein structure prediction, Structure, 30, 1169, 10.1016/j.str.2022.05.001 · DOI
  21. Sledzieski, S. et al. Democratizing Protein Language Models with Parameter-Efficient Fine-Tuning. Preprint at https://www.biorxiv.org/content/10.1101/2023.11.09.566187v1 (2023). · DOI · Paper page
  22. 2019, Evaluating Protein Transfer Learning with TAPE, Adv. neural information processing systems, 32, 9689
  23. Dallago, C. et al. FLIP: Benchmark tasks in fitness landscape inference for proteins. Preprint at https://www.biorxiv.org/content/10.1101/2021.11.09.467890v2 (2021). · DOI
  24. Feng, J. , Jiang, M. , Shih, J. & Chai, Q. Antibody apparent solubility prediction from sequence by transfer learning. Iscience 25 (2022). · DOI
  25. Leem, J. , Mitchell, L. S. , Farmery, J. H. , Barton, J. & Galson, J. D. Deciphering the language of antibodies using self-supervised learning. Patterns 3 (2022). · DOI
  26. 10.1002/pro.4497 · DOI · Paper page
  27. 10.1038/s41587 · DOI
  28. Groth, P. M. , Michael, R. , Salomon, J. , Tian, P. & Boomsma, W. FLOP: Tasks for Fitness Landscapes Of Protein wildtypes. Preprint at https://www.biorxiv.org/content/10.1101/2023.06.21.545880v1 (2023). · DOI
  29. 10.1007/s00439-021-02411-y · DOI
  30. Nijkamp, E. , Ruffolo, J. , Weinstein, E. N. , Naik, N. & Madani, A. ProGen2: Exploring the Boundaries of Protein Language Models. Preprint at https://arxiv.org/abs/2206.13517 (2022). · arXiv
  31. Notin, P. et al. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. Int. Conf. on Mach. Learn. 16990–17017 (2022).
  32. 2021, Protein embeddings and deep learning predict binding residues for various ligand classes, Sci. Reports, 11, 23916
  33. 1993, Prediction of protein secondary structure at better than 70% accuracy, J. molecular biology, 232, 584, 10.1006/jmbi.1993.1413 · DOI
  34. 10.1093/bioadv/vbab035 · DOI · Paper page
  35. Madani, A. et al. Large language models generate functional protein sequences across diverse families. Nat. Biotechnol. 1–8 (2023). · DOI · Paper page
  36. Heinzinger, M. et al. Bilingual Language Model for Protein Sequence and Structure. Preprint at https://www.biorxiv.org/content/10.1101/2023.07.23.550085v2 (2023). · DOI · Paper page
  37. 2022, AbLang: an antibody language model for completing antibody sequences, Bioinforma. Adv, 2, vbac046, 10.1093/bioadv/vbac046 · DOI · Paper page
  38. Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models. Preprint at https://arxiv.org/abs/2106.09685 (2021). · arXiv
  39. 2023, Parameter-efficient fine-tuning of large-scale pre-trained language models, Nat. Mach. Intell, 5, 220, 10.1038/s42256-023-00626-4 · DOI · Paper page
  40. 1999, Catastrophic forgetting in connectionist networks, Trends cognitive sciences, 3, 128, 10.1016/S1364-6613(99)01294-2 · DOI · Paper page
  41. 2017, Overcoming catastrophic forgetting in neural networks, Proc. national academy sciences, 114, 3521, 10.1073/pnas.1611835114 · DOI · Paper page
  42. 10.18653/v1/P18-1031 · DOI · Paper page
  43. Wortsman, M. et al. Robust fine-tuning of zero-shot models. Proc. IEEE/CVF Conf. on Comput. Vis. Pattern Recognit. 7959–7971 (2022). · DOI · Paper page
  44. 2022, NetSolP: predicting protein solubility in Escherichia coli using language models, Bioinformatics, 38, 941, 10.1093/bioinformatics/btab801 · DOI
  45. Wang, D. , Fei, Y. E. & Zhou, H. On Pre-training Language Model for Antibody. The Eleventh Int. Conf. on Learn. Represent. (2022).
  46. 2023, TSignal: a transformer model for signal peptide prediction, Bioinformatics, 39, i347, 10.1093/bioinformatics/btad228 · DOI
  47. 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, The J. Mach. Learn. Res, 21, 5485
  48. Devlin, J. , Chang, M.-W. , Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Preprint at https://arxiv.org/abs/1810.04805 (2018). · arXiv
  49. Liu, S.-Y. et al. DoRA: Weight-Decomposed Low-Rank Adaptation. Preprint at https://arxiv.org/abs/2402.09353 (2024). · arXiv
  50. 2022, Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Adv. Neural Inf. Process. Syst, 35, 1950
  51. Li, X. L. & Liang, P. Prefix-Tuning: Optimizing Continuous Prompts for Generation. Preprint at https://arxiv.org/abs/2101.00190 (2021). · arXiv
  52. 2018, Assessment of hard target modeling in CASP12 reveals an emerging role of alignment-based contact prediction methods, Proteins: Struct. Funct. Bioinforma, 86, 97, 10.1002/prot.25423 · DOI
  53. 1994, Redefining the goals of protein secondary structure prediction, J. molecular biology, 235, 13, 10.1016/S0022-2836(05)80007-5 · DOI
  54. 2020, ODiNPred: comprehensive prediction of protein order and disorder, Sci. Reports, 10, 14780
  55. Nielsen, J. T. & Mulder, F. A. Quantitative Protein Disorder Assessment Using NMR Chemical Shifts. Intrinsically Disord. proteins: methods protocols 303–317 (2020). · DOI
  56. 10.1038/nbt.3988 · DOI · Paper page
  57. Park, Y. , Metzger, B. P. & Thornton, J. W. The simplicity of protein sequence-function relationships. Preprint at https://www.biorxiv.org/content/10.1101/2023.09.02.556057v1 (2023). · DOI
  58. 10.1016/S0969-2126(01)00157-5 · DOI
  59. 2017, Efficacy and safety of voretigene neparvovec (AAV2-hRPE65v2) in patients with RPE65-mediated inherited retinal dystrophy: a randomised, controlled, open-label, phase 3 trial, The Lancet, 390, 849, 10.1016/S0140-6736(17)31868-8 · DOI · Paper page
  60. 10.1126/science.8303295 · DOI · Paper page
  61. van Kempen, M. et al. Fast and accurate protein structure search with foldseek. Nat. Biotechnol. 1–4 (2023). · DOI · Paper page
  62. Li, F.-Z. , Amini, A. P. , Yang, K. K. & Lu, A. X. Pretrained protein language model transfer learning: is the final layer representation what we want. Proc. Mach. Learn. for Struct. Biol. Work. NeurIPS 2022 (2022).
  63. Valeriani, L. et al. The geometry of hidden representations of large transformer models. Preprint at https://arxiv.org/abs/2302.00294 (2023). · DOI
  64. Liu, Y. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Preprint at https://arxiv.org/abs/1907.11692 (2019). · arXiv
  65. 10.1038/nmeth.3027 · DOI · Paper page
  66. 10.1038/nature17995 · DOI · Paper page
  67. 2021, Deep diversification of an AAV capsid protein by machine learning, Nat. Biotechnol, 39, 691, 10.1038/s41587-020-00793-4 · DOI
  68. 10.7554/eLife.16965 · DOI
  69. 10.1126/science.aan0693 · DOI · Paper page
  70. 10.1038/s41592-020-0801-4 · DOI
  71. 10.1093/bioinformatics/btx431 · DOI
  72. 2019, NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning, Proteins: Struct. Funct. Bioinforma, 87, 520, 10.1002/prot.25674 · DOI
  73. Kingma, D. P. & Ba, J. Adam: Adam: A Method for Stochastic Optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014). · arXiv

Metadata

Authors
Schmirler, R., Heinzinger, M., & Rost, B.
Affiliations
  • AbbVie (Germany) (DE)
  • Technical University of Munich (DE)
Corresponding author
Schmirler, Robert
ORCID
Venue
Nature Communications
Publisher
openRxiv
Published
2023-12-14
Volume
15
Issue
1
Pages
7407-7407
ISSN
2041-1723 (ISSN-L)
DOI
10.1101/2023.12.13.571462
PMC
PMC11358375
OpenAlex
W4401966068
Cited by
229
References
1
Citations by year
2026: 107 · 2025: 106 · 2024: 16
FWCI
31.17
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
Funders
  • Alexander von Humboldt-Stiftung
  • AbbVie
  • Deutsche Forschungsgemeinschaft (DFG-GZ: RO1320/4-1, RO1320/4‐1)
  • Bundesministerium für Bildung und Forschung
  • Technische Universität München
  • AbbVie Deutschland
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
Keywords
protein language models, fine-tuning, ESM-2, ProtT5, ankh, parameter-efficient fine-tuning, protein property prediction, protein embeddings
MeSH terms
Algorithms, Natural Language Processing, Proteins, Computational Biology, Databases, Protein

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review