Protein Language Models: Is Scaling Necessary?
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Public protein sequence databases contain samples from the fitness landscape explored by nature. Protein language models (pLMs) pre-trained on these sequences aim to capture this landscape for tasks like property prediction and protein design. Following the same trend as in natural language processing, pLMs have continuously been scaled up. However, the premise that scale leads to better performance assumes that source databases provide an accurate representation of the underlying fitness landscape, which is likely false. By developing an efficient codebase, designing a modern architecture, and addressing data quality concerns such as sample bias, we introduce AMPLIFY, a best-in-class pLM that is orders of magnitude less expensive to train and deploy than previous models. Furthermore, to support the scientific community and democratize the training of pLMs, we have open-sourced AMPLIFY’s pre-training codebase, data, and model checkpoints.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Accurate structure prediction of biomolecular interactions with AlphaFold 3
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. (2024) · Nature · doi:10.1038/s41586-024-07487-w · Cited by 15,861 · cited 1×
“…in using machine learning approaches to address protein structure prediction and design have emerged as a direct consequence of this shift (Senior et al., 2020; Jumper et al., 2021; Dauparas et al., 2022; Wang et al., 2022; Ingraham et al., 2023; Watson et al., 2023; Abramson et al., 2024).”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“This scaling is motivated by the recent achievements of larger networks in natural language processing, where their language modeling abilities have been shown to scale as a power law of the model size given a dataset and compute budget (Kaplan et al., 2020; Hoffmann et al., 2022).”
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 2×
“…databases avoid the biased sampling inherent to structure databases, and by approaching protein sequence as a natural language problem researchers have demonstrated that protein language models (pLMs) can learn structural information independently of structure databases (Rives et al., 2021).”
Coactivator condensation at super-enhancers links phase separation and gene control
Sabari, B. R., Dall’Agnese, A., Boija, A., Klein, I. A., Coffey, E. L., Shrinivas, K., Abraham, B. J., Hannett, N. M., Zamudio, A. V., Manteiga, J. C., et al. (2018) · Science · doi:10.1126/science.aar3958 · Cited by 2,595
Research works printed but not linked: 45
- 10.1126/science.181.4096.223 · DOI · Paper page
- 10.1002/anie.201708408 · DOI · Paper page
- 10.1038/75556 · DOI · Paper page
- 10.1093/nar/gkad928 · DOI · Paper page
- 10.1038/nrm.2017.7 · DOI · Paper page
- Nicholas Carlini , Daphne Ippolito , Matthew Jagielski , Katherine Lee , Florian Tramer , and Chiyuan Zhang . Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TatRHT_1cK.
- 10.1093/nar/gkab1054 · DOI · Paper page
- 10.1093/nar/gkac1052 · DOI · Paper page
- Dimiter S Dimitrov . Therapeutic proteins. Therapeutic Proteins: Methods and Protocols, pages 1–26, 2012. · DOI · Paper page
- 10.1093/bioinformatics/14.9.755 · DOI · Paper page
- 2022, Controllable protein design with language models, Nature Machine Intelligence, 4, 521, 10.1038/s42256-022-00499-z · DOI · Paper page
- Leo Gao , Stella Biderman , Sid Black , Laurence Golding , Travis Hoppe , Charles Foster , Jason Phang , Horace He , Anish Thite , Noa Nabeshima , et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- Alicia Golden , Samuel Hsia , Fei Sun , Bilge Acun , Basil Hosmer , Yejin Lee , Zachary DeVito , Jeff Johnson , Gu-Yeon Wei , David Brooks , et al. Is flash attention stable? arXiv preprint arXiv:2405.02803, 2024. · arXiv
- 10.1101/2024.07.01.600583 · doi:10.1101/2024.07.01.600583
- Jordan Hoffmann , Sebastian Borgeaud , Arthur Mensch , Elena Buchatskaya , Trevor Cai , Eliza Rutherford , Diego de Las Casas , Lisa Anne Hendricks , Johannes Welbl , Aidan Clark , et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
- Chloe Hsu , Robert Verkuil , Jason Liu , Zeming Lin , Brian Hie , Tom Sercu , Adam Lerer , and Alexander Rives . Learning inverse folding from millions of predicted structures. In Kamalika Chaudhuri , Stefanie Jegelka, Le Song , Csaba Szepesvari , Gang Niu , and Sivan Sabato , editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8946–8970. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/hsu22a.html.
- 10.1002/bip.360221211 · DOI · Paper page
- 2011, Account-ing for conformational entropy in predicting binding free energies of protein-protein interactions, Proteins: Structure, Function, and Bioinformatics, 79, 444, 10.1002/prot.22894 · DOI
- 10.1186/s12859-018-2367-z · DOI
- 10.1038/s41580-020-0272-6 · DOI · Paper page
- 2000, The consensus concept for thermostability engineering of proteins, Biochimica et Biophysica Acta (BBA)-protein structure and molecular enzymology, 2, 408
- 2020, Limits to depth efficiencies of self-attention, Advances in Neural Information Processing Systems, 33, 22640
- 2022, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction, BioRxiv, 2022, 500902
- 10.1126/science.ade2574 · doi:10.1126/science.ade2574 · Paper page
- I Loshchilov . Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
- 10.1038/s41580-022-00566-8 · DOI · Paper page
- 2010, Protein dynamics and conformational disorder in molecular recognition, Journal of Molecular Recognition: An Interdisciplinary Journal, 23, 105, 10.1002/jmr.961 · DOI · Paper page
- 2011, Critical assessment of methods of protein structure prediction (casp)—round ix. Proteins: Structure, Function, and Bioinformatics, 79, 1, 10.1002/prot.23200 · DOI · Paper page
- 2014, Critical assessment of methods of protein structure prediction (casp)—round x. Proteins: Structure, Function, and Bioinformatics, 82, 1, 10.1002/prot.24452 · DOI · Paper page
- 2013, Molecular assembly and mechanical properties of the extracellular matrix: A fibrous protein perspective, Biochimica et Biophysica Acta (BBA)-Molecular Basis of Disease, 1832, 866
- 2021, Deep learning on a data diet: Finding important examples early in training, Advances in neural information processing systems, 34, 20596
- Felipe Garcia Quiroz , Vincent F Fiore , John Levorse , Lisa Polak , Ellen Wong , H Amalia Pasolli , and Elaine Fuchs . Liquid-liquid phase separation drives skin barrier formation. Science, 367(6483):eaax9554, 2020. · DOI · Paper page
- Mathieu Seppey , Mosè Manni , and Evgeny M Zdobnov . Busco: assessing genome assembly and annotation completeness. Gene prediction: methods and protocols, pages 227–245, 2019. · DOI · Paper page
- Mary Shelley . Frankenstein: the 1818 text. Penguin, 2018. · doi:10.1093/owc/9780198840824.001.0001
- Yongdae Shin and Clifford P Brangwynne . Liquid phase condensation in cell physiology and disease. Science, 357(6357):eaaf4382, 2017. · DOI · Paper page
- 10.1016/0022-2836(73)90011-9 · DOI · Paper page
- 10.1093/bioinformatics/btv351 · DOI · Paper page
- Boris Steipe , Britta Schiller , Andreas Plückthun , and Stefan Steinbacher . Sequence statistics reliably predict stabilizing mutations in a protein domain, 1994. · DOI · Paper page
- 10.1016/j.molcel.2013.05.021 · DOI · Paper page
- Hugo Touvron , Louis Martin , Kevin Stone , Peter Albert , Amjad Almahairi , Yasmine Babaei , Nikolay Bashlykov , Soumya Batra , Prajjwal Bhargava , Shruti Bhosale , et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
- 10.1093/nar/gku1028 · DOI · Paper page
- 10.1371/journal.pone.0022060 · DOI
- 10.1016/j.sbi.2021.01.008 · DOI · Paper page
- 10.1038/s41577-021-00572-5 · DOI · Paper page
- Ruibin Xiong , Yunchang Yang, Di He , Kai Zheng , Shuxin Zheng , Chen Xing , Huishuai Zhang , Yanyan Lan , Liwei Wang , and Tie-Yan Liu . On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
Cited by
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S. P., Hawkins-Hooker, A., Ektefaie, Y., et al. (2024) · doi:10.1101/2024.10.02.616248 · Cited by 37
BAGEL: Protein engineering via exploration of an energy landscape
(2025) · doi:10.1371/journal.pcbi.1013774 · Cited by 7
Metadata
- Authors
- Fournier, Q., Vernon, R. M., van der Sloot, A., Schulz, B., Chandar, S., & Langmead, C. J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2024-09-23
- OpenAlex
- W4402821735
- Cited by
- 26
- References
- 40
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Protein Language Models: Is Scaling Necessary? https://app.cochat.ai/papers/10.1101/2024.09.23.614603
- The paper
- Fournier, Q., Vernon, R. M., van der Sloot, A., Schulz, B., Chandar, S., & Langmead, C. J. (2024). Protein Language Models: Is Scaling Necessary? bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2024.09.23.614603
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.