The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
ProGen2: Exploring the boundaries of protein language models
Open accessAbstract
Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial-intelligence-driven protein design. However, we lack a sufficient understanding of how very large-scale models and data play a role in effective protein model development. We introduce a suite of protein language models, named ProGen2, that are scaled up to 6.4B parameters and trained on different sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases. ProGen2 models show state-of-the-art performance in capturing the distribution of observed evolutionary sequences, generating novel viable sequences, and predicting protein fitness without additional fine-tuning. As large model sizes and raw numbers of protein sequences continue to become more widely accessible, our results suggest that a growing emphasis needs to be placed on the data distribution provided to a protein sequence model. Our models and code are open sourced for widespread adoption in protein engineering. A record of this paper's Transparent Peer Review process is included in the supplemental information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Evo also reaches competitive performance with leading protein-specific language models (41, 46–48) (Fig. 2B).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Fast and accurate protein structure search with Foldseek
van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L. M., Söding, J., & Steinegger, M. (2022) · doi:10.1101/2022.02.07.479398 · Cited by 2,598
Fast and accurate protein structure search with Foldseek
van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L. M., Söding, J., & Steinegger, M. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01773-0 · Cited by 2,596
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928
Leaver-Fay (2011) · doi:10.1016/b978-0-12-381270-4.00019-6 · Cited by 1,904
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078
Self-citationTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
Clustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Disease variant prediction with deep generative models of evolutionary data
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., & Marks, D. S. (2021) · Nature · doi:10.1038/s41586-021-04043-8 · Cited by 791
Mutation effects predicted from sequence co-variation
Hopf, T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. P. I., Springer, M., Sander, C., & Marks, D. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3769 · Cited by 775
PyRosetta: a script-based interface for implementing molecular modeling algorithms using Rosetta
Chaudhury, S., Lyskov, S., & Gray, J. J. (2010) · Bioinform. · doi:10.1093/bioinformatics/btq007 · Cited by 772
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719
Arnold, F. H. (1998) · Accounts of Chemical Research · doi:10.1021/ar960017f · Cited by 706
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601
CATH: increased structural coverage of functional space
Sillitoe, I., Bordin, N., Dawson, N., Waman, V. P., Ashford, P., Scholes, H. M., Pang, C. S. M., Woodridge, L., Rauer, C., Sen, N., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa1079 · Cited by 509
The CamSol method of rational design of protein mutants with enhanced solubility
Sormanni, P., Aprile, F. A., & Vendruscolo, M. (2015) · Journal of Molecular Biology · doi:10.1016/j.jmb.2014.09.026 · Cited by 494
Efficient evolution of human antibodies from general protein language models
Hie, B., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P. A. B., Tang, S., Wu, W. C. H., Pak, J. E., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 472
ANARCI: antigen receptor numbering and receptor classification
Dunbar, J., & Deane, C. M. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv552 · Cited by 421
Expanding functional protein sequence spaces using generative adversarial networks
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00310-5 · Cited by 388
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 383
Self-citationImproving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 364
Self-citationFive computational developability guidelines for therapeutic antibody profiling
Raybould, M. I. J., Marks, C., Krawczyk, K., Taddese, B., Nowak, J., Lewis, A. P., Bujotzek, A., Shi, J., & Deane, C. M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1810576116 · Cited by 357
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
Self-citationProtein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320
Olsen, T. H., Boyles, F., & Deane, C. M. (2022) · Protein Science · doi:10.1002/pro.4205 · Cited by 319
An evolution-based model for designing chorismate mutase enzymes
Russ, W. P., Figliuzzi, M., Stocker, C., Barrat-Charlaix, P., Socolich, M., Kast, P., Hilvert, D., Monasson, R., Cocco, S., Weigt, M., et al. (2020) · Science · doi:10.1126/science.aba3304 · Cited by 313
Engineering highly functional thermostable proteins using ancestral sequence reconstruction
Gumulya, Y., Baek, J., Wun, S. J., Thomson, R. E. S., Harris, K. L., Hunter, D. J. B., Behrendorff, J. B. Y. H., Kulig, J. K., Zheng, S., Wu, X., et al. (2018) · Nature Catalysis · doi:10.1038/s41929-018-0159-5 · Cited by 201
Prediction of aggregation prone regions of therapeutic proteins
Chennamsetty, N., Voynov, V., Kayser, V., Helk, B., & Trout, B. L. (2010) · The Journal of Physical Chemistry B · doi:10.1021/jp911706q · Cited by 200
AbLang: an antibody language model for completing antibody sequences
Olsen, T. H., Moal, I. H., & Deane, C. M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac046 · Cited by 198
Gene3D: Extensive prediction of globular domains in proteins
Lewis, T. E., Sillitoe, I., Dawson, N., Lam, S., Clarke, T., Lee, D. A., Orengo, C., & Lees, J. (2017) · Nucleic Acids Res. · doi:10.1093/nar/gkx1069 · Cited by 195
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 191
Self-citationDeciphering antibody affinity maturation with language models and weakly supervised learningPreprint
Ruffolo, J. A., Gray, J. J., & Sulam, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.07782 · Cited by 190
Self-citationConvolutions are competitive with transformers for protein sequence pretrainingPreprint
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · bioRxiv · doi:10.1101/2022.05.19.492714 · Cited by 172
Deciphering the language of antibodies using self-supervised learning
Leem, J., Mitchell, L., Farmery, J. H. R., Barton, J., & Galson, J. D. (2022) · Patterns · doi:10.1016/j.patter.2022.100513 · Cited by 169
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 142
Self-citationConvolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137
RITA: a Study on Scaling Up Generative Protein Sequence ModelsPreprint
Hesslow, D., Zanichelli, N., Notin, P., Poli, I., & Marks, D. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05789 · Cited by 125
Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A., Marks, D. S., & Gal, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.13760 · Cited by 118
Warszawski, S., Katz, A. B., Lipsh‐Sokolik, R., Khmelnitsky, L., Ben‐Nissan, G., Javitt, G., Dym, O., Unger, T., Knop, O., Albeck, S., et al. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007207 · Cited by 114
Koenig, P., Lee, C. V., Walters, B. T., Janakiraman, V., Stinson, J., Patapoff, T. W., & Fuh, G. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1613231114 · Cited by 107
Pretrained Transformers as Universal Computation EnginesPreprint
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05247 · Cited by 95
Generative language modeling for antibody designPreprint
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.12.13.472419 · Cited by 49
Self-citationProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42
Self-citationNon-Identifiability and the Blessings of Misspecification in Models of Molecular Fitness
Weinstein, E. N., Amin, A., Frazer, J., & Marks, D. S. (2022) · neural information processing systems · doi:10.52202/068431-0396 · Cited by 21
Self-citationNon-identifiability and the Blessings of Misspecification in Models of Molecular FitnessPreprint
Weinstein, E. N., Amin, A. N., Frazer, J., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.01.29.478324 · Cited by 19
Self-citationJi, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Flagged first, then by citations
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Simulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Genome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 489
Metadata
- Authors
- Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A.
- Affiliations
- Salesforce (United States) (US)
- Johns Hopkins University (US)
- Columbia University (US)
- Corresponding author
- Madani, Ali
- Venue
- Cell Systems
- Publisher
- Elsevier BV
- Published
- 2023-11
- Volume
- 14
- Issue
- 11
- Pages
- 968-978.e3
- ISSN
- 2405-4712 (ISSN-L)
- OpenAlex
- W4388024559
- Cited by
- 480
- References
- 57
- Citations by year
- 2026: 150 · 2025: 169 · 2024: 112 · 2023: 39
- FWCI
- 42.06
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 380 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Cell Systems (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, protein sequence generation, protein fitness prediction, protein design, model scaling, data distribution, immune repertoire, attention-based models
- MeSH terms
- Amino Acid Sequence, Artificial Intelligence, Language, Proteins, Databases, Factual
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
