CoChat

Protein Language Models: Is Scaling Necessary?

Open access
Fournier, Quentin, Vernon, Robert M., van der Sloot, Almer, Schulz, Benjamin, Chandar, Sarath, Langmead, Christopher James · 2024 · bioRxiv (Cold Spring Harbor Laboratory)
PreprintCited by 26

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Abstract

Public protein sequence databases contain samples from the fitness landscape explored by nature. Protein language models (pLMs) pre-trained on these sequences aim to capture this landscape for tasks like property prediction and protein design. Following the same trend as in natural language processing, pLMs have continuously been scaled up. However, the premise that scale leads to better performance assumes that source databases provide an accurate representation of the underlying fitness landscape, which is likely false. By developing an efficient codebase, designing a modern architecture, and addressing data quality concerns such as sample bias, we introduce AMPLIFY, a best-in-class pLM that is orders of magnitude less expensive to train and deploy than previous models. Furthermore, to support the scientific community and democratize the training of pLMs, we have open-sourced AMPLIFY’s pre-training codebase, data, and model checkpoints.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.09.23.614603

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 40 references

Research works printed but not linked: 45

  1. 10.1126/science.181.4096.223 · DOI · Paper page
  2. 10.1002/anie.201708408 · DOI · Paper page
  3. 10.1038/75556 · DOI · Paper page
  4. 10.1093/nar/gkad928 · DOI · Paper page
  5. 10.1038/nrm.2017.7 · DOI · Paper page
  6. Nicholas Carlini , Daphne Ippolito , Matthew Jagielski , Katherine Lee , Florian Tramer , and Chiyuan Zhang . Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TatRHT_1cK.
  7. 10.1093/nar/gkab1054 · DOI · Paper page
  8. 10.1093/nar/gkac1052 · DOI · Paper page
  9. Dimiter S Dimitrov . Therapeutic proteins. Therapeutic Proteins: Methods and Protocols, pages 1–26, 2012. · DOI · Paper page
  10. 10.1093/bioinformatics/14.9.755 · DOI · Paper page
  11. 2022, Controllable protein design with language models, Nature Machine Intelligence, 4, 521, 10.1038/s42256-022-00499-z · DOI · Paper page
  12. Leo Gao , Stella Biderman , Sid Black , Laurence Golding , Travis Hoppe , Charles Foster , Jason Phang , Horace He , Anish Thite , Noa Nabeshima , et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
  13. Alicia Golden , Samuel Hsia , Fei Sun , Bilge Acun , Basil Hosmer , Yejin Lee , Zachary DeVito , Jeff Johnson , Gu-Yeon Wei , David Brooks , et al. Is flash attention stable? arXiv preprint arXiv:2405.02803, 2024. · arXiv
  14. 10.1101/2024.07.01.600583 · doi:10.1101/2024.07.01.600583
  15. Jordan Hoffmann , Sebastian Borgeaud , Arthur Mensch , Elena Buchatskaya , Trevor Cai , Eliza Rutherford , Diego de Las Casas , Lisa Anne Hendricks , Johannes Welbl , Aidan Clark , et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
  16. Chloe Hsu , Robert Verkuil , Jason Liu , Zeming Lin , Brian Hie , Tom Sercu , Adam Lerer , and Alexander Rives . Learning inverse folding from millions of predicted structures. In Kamalika Chaudhuri , Stefanie Jegelka, Le Song , Csaba Szepesvari , Gang Niu , and Sivan Sabato , editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8946–8970. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/hsu22a.html.
  17. 10.1002/bip.360221211 · DOI · Paper page
  18. 2011, Account-ing for conformational entropy in predicting binding free energies of protein-protein interactions, Proteins: Structure, Function, and Bioinformatics, 79, 444, 10.1002/prot.22894 · DOI
  19. 10.1186/s12859-018-2367-z · DOI
  20. 10.1038/s41580-020-0272-6 · DOI · Paper page
  21. 2000, The consensus concept for thermostability engineering of proteins, Biochimica et Biophysica Acta (BBA)-protein structure and molecular enzymology, 2, 408
  22. 2020, Limits to depth efficiencies of self-attention, Advances in Neural Information Processing Systems, 33, 22640
  23. 2022, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction, BioRxiv, 2022, 500902
  24. 10.1126/science.ade2574 · doi:10.1126/science.ade2574 · Paper page
  25. I Loshchilov . Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  26. 10.1038/s41580-022-00566-8 · DOI · Paper page
  27. 2010, Protein dynamics and conformational disorder in molecular recognition, Journal of Molecular Recognition: An Interdisciplinary Journal, 23, 105, 10.1002/jmr.961 · DOI · Paper page
  28. 2011, Critical assessment of methods of protein structure prediction (casp)—round ix. Proteins: Structure, Function, and Bioinformatics, 79, 1, 10.1002/prot.23200 · DOI · Paper page
  29. 2014, Critical assessment of methods of protein structure prediction (casp)—round x. Proteins: Structure, Function, and Bioinformatics, 82, 1, 10.1002/prot.24452 · DOI · Paper page
  30. 2013, Molecular assembly and mechanical properties of the extracellular matrix: A fibrous protein perspective, Biochimica et Biophysica Acta (BBA)-Molecular Basis of Disease, 1832, 866
  31. 2021, Deep learning on a data diet: Finding important examples early in training, Advances in neural information processing systems, 34, 20596
  32. Felipe Garcia Quiroz , Vincent F Fiore , John Levorse , Lisa Polak , Ellen Wong , H Amalia Pasolli , and Elaine Fuchs . Liquid-liquid phase separation drives skin barrier formation. Science, 367(6483):eaax9554, 2020. · DOI · Paper page
  33. Mathieu Seppey , Mosè Manni , and Evgeny M Zdobnov . Busco: assessing genome assembly and annotation completeness. Gene prediction: methods and protocols, pages 227–245, 2019. · DOI · Paper page
  34. Mary Shelley . Frankenstein: the 1818 text. Penguin, 2018. · doi:10.1093/owc/9780198840824.001.0001
  35. Yongdae Shin and Clifford P Brangwynne . Liquid phase condensation in cell physiology and disease. Science, 357(6357):eaaf4382, 2017. · DOI · Paper page
  36. 10.1016/0022-2836(73)90011-9 · DOI · Paper page
  37. 10.1093/bioinformatics/btv351 · DOI · Paper page
  38. Boris Steipe , Britta Schiller , Andreas Plückthun , and Stefan Steinbacher . Sequence statistics reliably predict stabilizing mutations in a protein domain, 1994. · DOI · Paper page
  39. 10.1016/j.molcel.2013.05.021 · DOI · Paper page
  40. Hugo Touvron , Louis Martin , Kevin Stone , Peter Albert , Amjad Almahairi , Yasmine Babaei , Nikolay Bashlykov , Soumya Batra , Prajjwal Bhargava , Shruti Bhosale , et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  41. 10.1093/nar/gku1028 · DOI · Paper page
  42. 10.1371/journal.pone.0022060 · DOI
  43. 10.1016/j.sbi.2021.01.008 · DOI · Paper page
  44. 10.1038/s41577-021-00572-5 · DOI · Paper page
  45. Ruibin Xiong , Yunchang Yang, Di He , Kai Zheng , Shuxin Zheng , Chen Xing , Huishuai Zhang , Yanyan Lan , Liwei Wang , and Tie-Yan Liu . On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.

Cited by

See all 26

Metadata

Authors
Fournier, Q., Vernon, R. M., van der Sloot, A., Schulz, B., Chandar, S., & Langmead, C. J.
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2024-09-23
OpenAlex
W4402821735
Cited by
26
References
40
Type
posted content
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Protein Language Models: Is Scaling Necessary? https://app.cochat.ai/papers/10.1101/2024.09.23.614603
The paper
Fournier, Q., Vernon, R. M., van der Sloot, A., Schulz, B., Chandar, S., & Langmead, C. J. (2024). Protein Language Models: Is Scaling Necessary? bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2024.09.23.614603

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.