Deep neural language modeling enables functional protein generation across families
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Bypassing nature’s evolutionary trajectory, de novo protein generation—defined as creating artificial protein sequences from scratch—could enable breakthrough solutions for biomedical and environmental challenges. Viewing amino acid sequences as a language, we demonstrate that a deep learning-based language model can generate functional artificial protein sequences across families, akin to generating grammatically and semantically correct natural language sentences on diverse topics. Our protein language model is trained by simply learning to predict the next amino acid for over 280 million protein sequences from thousands of protein families, without biophysical or coevolutionary modeling. We experimentally evaluate model-generated artificial proteins on five distinct antibacterial lysozyme families. Artificial proteins show similar activities and catalytic efficiencies as representative natural lysozymes, including hen egg white lysozyme, while reaching as low as 44% identity to any known naturally-evolved protein. The X-ray crystal structure of an enzymatically active artificial protein recapitulates the conserved fold and positioning of active site residues found in natural proteins. We demonstrate our language model’s ability to be adapted to different protein families by accurately predicting the functionality of artificial chorismate mutase and malate dehydrogenase proteins. These results indicate that neural language models successfully perform de novo protein generation across protein families and may prove to be a tool to shortcut evolution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 1×
“In contrast to transformer-based language models that encode amino acid sequences for discriminative protein prediction tasks 25,33,34 , ProGen is a decoder transformer tailored for autoregressive generation.”
ResultProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · cited 1×
“…proteins are useful for protein informatics tasks such as stability prediction, remote homology detection, and secondary structure prediction 10,24–26 , we show that the latest advances in deep learning-based language modeling can be adopted to generate artificial protein sequences, from scratch,…”
BackgroundThe coming of age of de novo protein design
Huang, P.-S., Boyken, S. E., & Baker, D. (2016) · Nature · doi:10.1038/nature19946 · Cited by 1,478
Terwilliger, T. C., Grosse-Kunstleve, R. W., Afonine, P. V., Moriarty, N. W., Zwart, P. H., Hung, L.-W., Read, R. J., & Adams, P. D. (2008) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s090744490705024x · Cited by 1,462
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · cited 1×
“In contrast to transformer-based language models that encode amino acid sequences for discriminative protein prediction tasks 25,33,34 , ProGen is a decoder transformer tailored for autoregressive generation.”
ResultCTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Self-citationEvaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 2×
“The discriminator architecture was a fine-tuned TAPE-BERT 34 .”
ResultOvchinnikov, S., Kamisetty, H., & Baker, D. (2014) · eLife · doi:10.7554/elife.02030 · Cited by 629
Principles for designing ideal protein structures
Koga, N., Tatsumi-Koga, R., Liu, G., Xiao, R., Acton, T. B., Montelione, G. T., & Baker, D. (2012) · Nature · doi:10.1038/nature11600 · Cited by 564
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508
Low-N protein engineering with data-efficient deep learning
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., & Church, G. M. (2021) · Nature Methods · doi:10.1038/s41592-021-01100-y · Cited by 422
Expanding functional protein sequence spaces using generative adversarial networks
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00310-5 · Cited by 388
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
Self-citationProtein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320
An evolution-based model for designing chorismate mutase enzymes
Russ, W. P., Figliuzzi, M., Stocker, C., Barrat-Charlaix, P., Socolich, M., Kast, P., Hilvert, D., Monasson, R., Cocco, S., Weigt, M., et al. (2020) · Science · doi:10.1126/science.aba3304 · Cited by 313
Protein structure predictions to atomic accuracy with AlphaFold
Jumper, J., & Hassabis, D. (2022) · Nature Methods · doi:10.1038/s41592-021-01362-6 · Cited by 265
Lessons from the lysozyme of phage T4
Baase, W. A., Liu, L., Tronrud, D. E., & Matthews, B. W. (2010) · Protein Science · doi:10.1002/pro.344 · Cited by 199
Mchaourab, H. S., Oh, K. J., Fang, C. J., & Hubbell, W. L. (1997) · Biochemistry · doi:10.1021/bi962114m · Cited by 198
Research works printed but not linked: 45
- 10.1073/pnas.1509508112 · DOI · Paper page
- 10.1038/nchembio.1966 · DOI · Paper page
- 10.1126/science.aad8865 · DOI · Paper page
- 10.1093/bioinformatics/bty862 · DOI · Paper page
- 2020, Signal Peptides Generated by Attention-Based Neural Networks, ACS Synth. Biol, 9, 2154, 10.1021/acssynbio.0c00219 · DOI · Paper page
- 2021, Deep diversification of an AAV capsid protein by machine learning, Nat. Biotechnol, 39, 691, 10.1038/s41587-020-00793-4 · DOI · Paper page
- 2021, Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations, Nat Biomed Eng, 5, 613, 10.1038/s41551-021-00689-x · DOI · Paper page
- 10.1093/bioinformatics/bth191 · DOI · Paper page
- 10.1093/nar/gni148 · DOI · Paper page
- 10.1093/nar/gkt1223 · DOI · Paper page
- Vaswani, A. et al. Attention Is All You Need. arXiv [cs.CL] (2017).
- Zellers, R. et al. Defending Against Neural Fake News. arXiv [cs.CL] (2019).
- 10.2307/412586 · DOI · Paper page
- Poplack, S. Sometimes I’ll start a sentence in Spanish Y TERMINO EN ESPAÑOL: toward a typology of code-switching1. Linguistics vol. 18 (1980). · DOI · Paper page
- Madani, A. et al. ProGen: Language Modeling for Protein Generation. arXiv [q-bio.BM] (2020). · DOI · Paper page
- 10.1162/153244303322533223 · DOI · Paper page
- 10.21203/rs.2.13774/v1 · DOI
- 10.1093/nar/gkr1178 · DOI · Paper page
- 10.1016/0005-2795(75)90109-9 · DOI · Paper page
- 2018, Catalytic diversity and cell wall binding repeats in the phage-encoded endolysins, Mol. Microbiol, 110, 879, 10.1111/mmi.14134 · DOI
- 2020, On the catalytic mechanism of bacteriophage endolysins: Opportunities for engineering, Biochim. Biophys. Acta: Proteins Proteomics, 1868, 140302, 10.1016/j.bbapap.2019.140302 · DOI
- Martin, P. P. Potts Models And Related Problems In Statistical Mechanics. (World Scientific, 1991). · DOI · Paper page
- 2015, Inferring Pairwise Interactions from Biological Data Using Maximum-Entropy Probability Models, PLoS Comput. Biol, 11, e1004182, 10.1371/journal.pcbi.1004182 · DOI · Paper page
- Goodfellow, I. J. et al. Generative Adversarial Networks. arXiv [stat.ML] (2014).
- Koehn, P. Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models. in Machine Translation: From Real Users to Research 115–124 (Springer Berlin Heidelberg, 2004). · DOI · Paper page
- 10.1093/bioinformatics/btab184 · DOI · Paper page
- 2009, Evaluation at atomic resolution of the role of strain in destabilizing the temperature-sensitive T4 lysozyme mutant Arg 96 → His, Protein Science, 18, 863, 10.1002/pro.93 · DOI
- 10.1126/science.8266098 · DOI · Paper page
- 10.1093/nar/gkv360 · DOI · Paper page
- 10.1101/846279 · DOI
- Ruder, S. , Peters, M. E. , Swayamdipta, S. & Wolf, T. Transfer Learning in Natural Language Processing. in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials 15–18 (Association for Computational Linguistics, 2019). · DOI · Paper page
- Huh, M. , Agrawal, P. & Efros, A. A. What makes ImageNet good for transfer learning? arXiv [cs.CV] (2016).
- The IUPAC stability constants database. Chem. Int. 28, (2006). · doi:10.1515/ci.2006.28.5.14
- 10.1038/75556 · DOI · Paper page
- Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization. arXiv [cs.LG] (2014).
- 2013, in Proceedings of the 30th International Conference on Machine Learning (eds, 28, 1310
- 10.1214/12-AOS1000. 1102.4807 · DOI · Paper page
- Holtzman, A. , Buys, J. , Du, L. , Forbes, M. & Choi, Y. The Curious Case of Neural Text Degeneration. in Eighth International Conference on Learning Representations (2020).
- Sun, Z. Z. et al. Protocols for implementing an Escherichia coli based TX-TL cell-free expression system for synthetic biology. J. Vis. Exp. e50762 (2013). · DOI · Paper page
- 10.1107/S0907444909047337 · DOI · Paper page
- 10.1107/S0021889807021206 · DOI · Paper page
- 10.1107/S2059798318000979 · DOI · Paper page
- 2020, Current approaches for automated model building into cryo-EM maps using Buccaneer with CCP-EM, Acta Crystallogr D Struct Biol, 76, 531, 10.1107/S2059798320005513 · DOI · Paper page
- 10.1107/S0907444910007493 · DOI · Paper page
- 10.1107/S0907444912001308 · DOI · Paper page
Cited by
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Scaffolding protein functional sites using deep learning
Wang, J., Lisanza, S., Juergens, D., Tischer, D., Watson, J. L., Castro, K. M., Ragotte, R., Saragovi, A., Milles, L. F., Baek, M., et al. (2022) · Science · doi:10.1126/science.abn2100 · Cited by 487
Learning inverse folding from millions of predicted structures
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., & Rives, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 416
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181
Metadata
- Authors
- Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, J. S., & Naik, N.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2021-07-18
- OpenAlex
- W3186612807
- Cited by
- 41
- References
- 25
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.