Deep neural language modeling enables functional protein generation across families
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Bypassing nature’s evolutionary trajectory, de novo protein generation—defined as creating artificial protein sequences from scratch—could enable breakthrough solutions for biomedical and environmental challenges. Viewing amino acid sequences as a language, we demonstrate that a deep learning-based language model can generate functional artificial protein sequences across families, akin to generating grammatically and semantically correct natural language sentences on diverse topics. Our protein language model is trained by simply learning to predict the next amino acid for over 280 million protein sequences from thousands of protein families, without biophysical or coevolutionary modeling. We experimentally evaluate model-generated artificial proteins on five distinct antibacterial lysozyme families. Artificial proteins show similar activities and catalytic efficiencies as representative natural lysozymes, including hen egg white lysozyme, while reaching as low as 44% identity to any known naturally-evolved protein. The X-ray crystal structure of an enzymatically active artificial protein recapitulates the conserved fold and positioning of active site residues found in natural proteins. We demonstrate our language model’s ability to be adapted to different protein families by accurately predicting the functionality of artificial chorismate mutase and malate dehydrogenase proteins. These results indicate that neural language models successfully perform de novo protein generation across protein families and may prove to be a tool to shortcut evolution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 1×
“In contrast to transformer-based language models that encode amino acid sequences for discriminative protein prediction tasks 25,33,34 , ProGen is a decoder transformer tailored for autoregressive generation.”
ResultProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · cited 1×
“…proteins are useful for protein informatics tasks such as stability prediction, remote homology detection, and secondary structure prediction 10,24–26 , we show that the latest advances in deep learning-based language modeling can be adopted to generate artificial protein sequences, from scratch,…”
BackgroundThe coming of age of de novo protein design
Huang, P.-S., Boyken, S. E., & Baker, D. (2016) · Nature · doi:10.1038/nature19946 · Cited by 1,478
Terwilliger, T. C., Grosse-Kunstleve, R. W., Afonine, P. V., Moriarty, N. W., Zwart, P. H., Hung, L.-W., Read, R. J., & Adams, P. D. (2008) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s090744490705024x · Cited by 1,462
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434
Research works printed but not linked: 45
- 10.1073/pnas.1509508112 · DOI · Paper page
- 10.1038/nchembio.1966 · DOI · Paper page
- 10.1126/science.aad8865 · DOI · Paper page
- 10.1093/bioinformatics/bty862 · DOI · Paper page
- 2020, Signal Peptides Generated by Attention-Based Neural Networks, ACS Synth. Biol, 9, 2154, 10.1021/acssynbio.0c00219 · DOI · Paper page
- 2021, Deep diversification of an AAV capsid protein by machine learning, Nat. Biotechnol, 39, 691, 10.1038/s41587-020-00793-4 · DOI · Paper page
- 2021, Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations, Nat Biomed Eng, 5, 613, 10.1038/s41551-021-00689-x · DOI · Paper page
- 10.1093/bioinformatics/bth191 · DOI · Paper page
- 10.1093/nar/gni148 · DOI · Paper page
- 10.1093/nar/gkt1223 · DOI · Paper page
- Vaswani, A. et al. Attention Is All You Need. arXiv [cs.CL] (2017).
- Zellers, R. et al. Defending Against Neural Fake News. arXiv [cs.CL] (2019).
- 10.2307/412586 · DOI · Paper page
- Poplack, S. Sometimes I’ll start a sentence in Spanish Y TERMINO EN ESPAÑOL: toward a typology of code-switching1. Linguistics vol. 18 (1980). · DOI · Paper page
- Madani, A. et al. ProGen: Language Modeling for Protein Generation. arXiv [q-bio.BM] (2020). · DOI · Paper page
- 10.1162/153244303322533223 · DOI · Paper page
- 10.21203/rs.2.13774/v1 · DOI
- 10.1093/nar/gkr1178 · DOI · Paper page
- 10.1016/0005-2795(75)90109-9 · DOI · Paper page
- 2018, Catalytic diversity and cell wall binding repeats in the phage-encoded endolysins, Mol. Microbiol, 110, 879, 10.1111/mmi.14134 · DOI
- 2020, On the catalytic mechanism of bacteriophage endolysins: Opportunities for engineering, Biochim. Biophys. Acta: Proteins Proteomics, 1868, 140302, 10.1016/j.bbapap.2019.140302 · DOI
- Martin, P. P. Potts Models And Related Problems In Statistical Mechanics. (World Scientific, 1991). · DOI · Paper page
- 2015, Inferring Pairwise Interactions from Biological Data Using Maximum-Entropy Probability Models, PLoS Comput. Biol, 11, e1004182, 10.1371/journal.pcbi.1004182 · DOI · Paper page
- Goodfellow, I. J. et al. Generative Adversarial Networks. arXiv [stat.ML] (2014).
- Koehn, P. Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models. in Machine Translation: From Real Users to Research 115–124 (Springer Berlin Heidelberg, 2004). · DOI · Paper page
- 10.1093/bioinformatics/btab184 · DOI · Paper page
- 2009, Evaluation at atomic resolution of the role of strain in destabilizing the temperature-sensitive T4 lysozyme mutant Arg 96 → His, Protein Science, 18, 863, 10.1002/pro.93 · DOI
- 10.1126/science.8266098 · DOI · Paper page
- 10.1093/nar/gkv360 · DOI · Paper page
- 10.1101/846279 · DOI
- Ruder, S. , Peters, M. E. , Swayamdipta, S. & Wolf, T. Transfer Learning in Natural Language Processing. in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials 15–18 (Association for Computational Linguistics, 2019). · DOI · Paper page
- Huh, M. , Agrawal, P. & Efros, A. A. What makes ImageNet good for transfer learning? arXiv [cs.CV] (2016).
- The IUPAC stability constants database. Chem. Int. 28, (2006). · doi:10.1515/ci.2006.28.5.14
- 10.1038/75556 · DOI · Paper page
- Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization. arXiv [cs.LG] (2014).
- 2013, in Proceedings of the 30th International Conference on Machine Learning (eds, 28, 1310
- 10.1214/12-AOS1000. 1102.4807 · DOI · Paper page
- Holtzman, A. , Buys, J. , Du, L. , Forbes, M. & Choi, Y. The Curious Case of Neural Text Degeneration. in Eighth International Conference on Learning Representations (2020).
- Sun, Z. Z. et al. Protocols for implementing an Escherichia coli based TX-TL cell-free expression system for synthetic biology. J. Vis. Exp. e50762 (2013). · DOI · Paper page
- 10.1107/S0907444909047337 · DOI · Paper page
- 10.1107/S0021889807021206 · DOI · Paper page
- 10.1107/S2059798318000979 · DOI · Paper page
- 2020, Current approaches for automated model building into cryo-EM maps using Buccaneer with CCP-EM, Acta Crystallogr D Struct Biol, 76, 531, 10.1107/S2059798320005513 · DOI · Paper page
- 10.1107/S0907444910007493 · DOI · Paper page
- 10.1107/S0907444912001308 · DOI · Paper page
Cited by
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Scaffolding protein functional sites using deep learning
Wang, J., Lisanza, S., Juergens, D., Tischer, D., Watson, J. L., Castro, K. M., Ragotte, R., Saragovi, A., Milles, L. F., Baek, M., et al. (2022) · Science · doi:10.1126/science.abn2100 · Cited by 487
Learning inverse folding from millions of predicted structures
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., & Rives, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 416
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181
Metadata
- Authors
- Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, J. S., & Naik, N.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2021-07-18
- OpenAlex
- W3186612807
- Cited by
- 41
- References
- 24
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep neural language modeling enables functional protein generation across families. https://app.cochat.ai/papers/10.1101/2021.07.18.452833
- The paper
- Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, J. S., & Naik, N. (2021). Deep neural language modeling enables functional protein generation across families. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2021.07.18.452833
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.