The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Controllable protein design with language models
Open accessAbstract
The twenty-first century is presenting humankind with unprecedented environmental and medical challenges. The ability to design novel proteins tailored for specific purposes would potentially transform our ability to respond to these issues in a timely manner. Recent advances in the field of artificial intelligence are now setting the stage to make this goal achievable. Protein sequences are inherently similar to natural languages: amino acids arrange in a multitude of combinations to form structures that carry function, the same way as letters form words and sentences carry meaning. Accordingly, it is not surprising that, throughout the history of natural language processing (NLP), many of its techniques have been applied to protein research problems. In the past few years we have witnessed revolutionary breakthroughs in the field of NLP. The implementation of transformer pre-trained models has enabled text generation with human-like capabilities, including texts with specific properties such as style or subject. Motivated by its considerable success in NLP tasks, we expect dedicated transformers to dominate custom protein sequence generation in the near future. Fine-tuning pre-trained models on protein families will enable the extension of their repertoires with novel sequences that could be highly divergent but still potentially functional. The combination of control tags such as cellular compartment or function will further enable the controllable design of novel protein functions. Moreover, recent model interpretability methods will allow us to open the ‘black box’ and thus enhance our understanding of folding principles. Early initiatives show the enormous potential of generative language models to design functional sequences. We believe that using generative text models to create novel proteins is a promising and largely unexplored field, and we discuss its foreseeable impact on protein design. Both proteins and natural language are essentially based on a sequential code, but feature complex interactions at multiple scales, which can be useful when transferring machine learning models from one domain to another. In this Review, Ferruz and Höcker summarize recent advances in language models, such as transformers, and their application to protein design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Research works printed but not linked: 64
- Lechner, H., Ferruz, N. & Höcker, B. Strategies for designing non-natural enzymes and binders. Curr. Opin. Chem. Biol. 47, 67–76 (2018). · DOI
- Gainza, P., Nisonoff, H. M. & Donald, B. R. Algorithms for protein design. Curr. Opin. Struct. Biol. 39, 16–26 (2016). · DOI
- Ferruz, N. et al. Identification and analysis of natural building blocks for evolution-guided fragment-based protein design. J. Mol. Biol. 432, 3898–3914 (2020). · DOI
- W, E. et al. Molecular evolution of FOXP2, a gene involved in speech and language. Nature 418, 869–872 (2002). · DOI · Paper page
- Arena, S. et al. Emergence of multiple EGFR extracellular mutations during cetuximab treatment in colorectal cancer. Clin. Cancer Res. 21, 2157–2166 (2015). · DOI · Paper page
- Lindqvist, Y. & Schneider, G. Circular permutations of natural protein sequences: structural evidence. Curr. Opin. Struct. Biol. 7, 422–427 (1997). · DOI
- Huang, P. S. et al. De novo design of a four-fold symmetric TIM-barrel protein with atomic-level accuracy. Nat. Chem. Biol. 12, 29–34 (2016). · DOI · Paper page
- Freeman, M. R., Blumenfeld, H. K. & Marian, V. Phonotactic constraints are activated across languages in bilinguals. Front. Psychol. 7, 702 (2016). · DOI
- Göbel, U., Sander, C., Schneider, R. & Valencia, A. Correlated mutations and residue contacts in proteins. Proteins Struct. Funct. Bioinformatics 18, 309–317 (1994). · DOI · Paper page
- Rao, R. M. et al. MSA Transformer. In Proc. 38th International Conference on Machine Learning Vol. 139, 8844–8856 https://proceedings.mlr.press/v139/rao21a.html (MLR, 2021).
- Nguyen, K. A., Im Walde, S. S. & Vu, N. T. Distinguishing antonyms and synonyms in a pattern-based neural network. In Proc. 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017 Vol. 1, 76–85 (Association for Computational Linguistics, 2017). · DOI
- Zhou, G. & Su, J. Named entity recognition using an HMM-based chunk tagger. In Proc. 40th Annual Meeting of the Association for Computational Linguistics, ACL ’02 473–480 https://doi.org/10.3115/1073083.1073163 (Association for Computational Linguistics, 2001). · DOI · Paper page
- Karchin, R., Cline, M., Mandel-Gutfreund, Y. & Karplus, K. Hidden Markov models that use predicted local structure for fold recognition: alphabets of backbone geometry. Proteins Struct. Funct. Genet. 51, 504–514 (2003). · DOI · Paper page
- Yakhnenko, O., Silvescu, A. & Honavar, V. Discriminatively trained Markov model for sequence classification. In Proc. IEEE International Conference on Data Mining, ICDM 498–505 https://doi.org/10.1109/ICDM.2005.52 (IEEE, 2005). · DOI
- Nguyen Ba, A. N., Pogoutse, A., Provart, N. & Moses, A. M. NLStradamus: a simple hidden Markov model for nuclear localization signal prediction. BMC Bioinformatics 10, 202 (2009). · DOI · Paper page
- Söding, J. Protein homology detection by HMM-HMM comparison. Bioinformatics 21, 951–960 (2005). · DOI · Paper page
- Bengio, Y. et al. A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003).
- Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient estimation of word representations in vector space. In Proc. 1st International Conference on Learning Representations, ICLR 2013 (ICLR, 2013).
- Mikolov, T., Chen, K., Corrado, G. & Dean, J. Distributed representations of words and phrases and their compositionality. In Proc. 26th International Conference on Neural Information Processing Systems Vol. 2, 3111–3119 (ACM, 2013).
- Young, T., Hazarika, D., Poria, S. & Cambria, E. Recent trends in deep learning based natural language processing. IEEE Comput. Intell. Mag. 13, 55–75 (2017). · DOI · Paper page
- Yang, K. K., Wu, Z., Bedbrook, C. N. & Arnold, F. H. Learned protein embeddings for machine learning. Bioinformatics 34, 2642–2648 (2018). · DOI · Paper page
- Asgari, E. & Mofrad, M. R. K. Continuous distributed representation of biological sequences for deep proteomics and genomics. PLoS ONE 10, e0141287 (2015). · DOI · Paper page
- Wang, S., Weng, S., Ma, J. & Tang, Q. DeepCNF-D: predicting protein order/disorder regions by weighted deep convolutional neural fields. Int. J. Mol. Sci. 16, 17315–17330 (2015). · DOI
- Zeng, H., Edwards, M. D., Liu, G. & Gifford, D. K. Convolutional neural network architectures for predicting DNA-protein binding. Bioinformatics 32, i121–i127 (2016). · DOI · Paper page
- Hou, J., Adhikari, B. & Cheng, J. DeepSF: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics 34, 1295–1303 (2018). · DOI · Paper page
- Bahdanau, D., Cho, K. H. & Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proc. 3rd International Conference on Learning Representations, ICLR 2015 (ICLR, 2015).
- Radford, A., Jozefowicz, R. & Sutskever, I. Learning to generate reviews and discovering sentiment. Preprint at https://arxiv.org/abs/1704.01444 (2017). · arXiv
- Krause, B., Murray, I., Renals, S. & Lu, L. Multiplicative LSTM for sequence modelling. In Proc. 5th International Conference on Learning Representations, ICLR 2017 https://doi.org/10.48550/arxiv.1609.07959 (ICLR, 2016). · DOI
- Senior, A. W. et al. Improved protein structure prediction using potentials from deep learning. Nature 577, 706–710 (2020). · DOI · Paper page
- Vaswani, A. et al. Transformer: attention is all you need. In Advances in Neural Information Processing Systems Vol. 2017, 5999–6009 (NIPS, 2017).
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1 4171–4186 (Association for Computational Linguistics, 2018).
- Wang, A. & Cho, K. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proc. Workshop on Methods for Optimizing and Evaluating Neural Language Generation 30–36 (ACL, 2019). · doi:10.18653/v1/w19-2304
- Sun, C., Qiu, X., Xu, Y. & Huang, X. in Lecture Notes in Computer Science Vol. 11856, 194–206 (Springer, 2019). · DOI · Paper page
- Wolf, T. et al. ransformers: state-of-the-art natural language processing. in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 38-45 (2020).
- Yu, L. et al. Grammar of protein domain architectures. Proc. Natl Acad. Sci. USA 116, 3636–3645 (2019). · DOI
- Ferruz, N. & Höcker, B. Dreaming ideal protein structures. Nat. Biotechnol. https://doi.org/10.1038/s41587-021-01196-9 (2022). · DOI
- Huang, B. et al. A backbone-centred energy function of neural networks for protein design. Nature 602, 523–528 (2022). · DOI
- Castro, E. et al. Guided generative protein design using regularized transformers. Preprint at https://arxiv.org/abs/2201.09948 (2022). · arXiv
- Moffat, L., Kandathil, S. M. & Jones, D. T. Design in the DARK: learning deep generative models for de novo protein design. Preprint at https://www.biorxiv.org/content/10.1101/2022.01.27.478087v1 (2022). · DOI · Paper page
- Lee, J. S. & Hsiang, J. Patent claim generation by fine-tuning OpenAI GPT-2. World Pat. Inf. 62, 101983 (2020). · DOI
- Gligorijević, V. et al. Function-guided protein design by deep manifold sampling. in Neural Information Processing Systems (NeurIPS, 2021). · DOI
- Madani, A. et al. ProGen: language modeling for protein generation. Preprint at https://www.biorxiv.org/content/10.1101/2020.03.07.982272v2 (2020). · DOI · Paper page
- Madani, A. et al. Deep neural language modeling enables functional protein generation across families. Preprint at https://www.biorxiv.org/content/10.1101/2021.07.18.452833v1 (2021). · DOI
- Rembeza, E. & Engqvist, M. K. M. Experimental and computational investigation of enzyme functional annotations uncovers misannotation in the EC 1.1.3.15 enzyme class. PLoS Comput. Biol. 17, e1009446 (2021). · DOI
- Chang, Y. C. et al. COMBREX-DB: an experiment centered database of protein function: knowledge, predictions and knowledge gaps. Nucleic Acids Res. 44, D330–D335 (2016). · DOI · Paper page
- Schwaller, P., Gaudin, T., Lányi, D., Bekas, C. & Laino, T. ‘Found in Translation’: predicting outcomes of complex organic chemistry reactions using neural sequence-to-sequence models. Chem. Sci. 9, 6091–6098 (2018). · DOI · Paper page
- Schwaller, P. et al. Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. Chem. Sci. 11, 3316–3325 (2020). · DOI · Paper page
- Grechishnikova, D. Transformer neural network for protein-specific de novo drug generation as a machine translation problem. Sci. Rep. 11, 321 (2021). · DOI · Paper page
- Schwaller, P. et al. Mapping the space of chemical reactions using attention-based neural networks. Nat. Mach. Intell. 3, 144–152 (2021). · DOI · Paper page
- Danilevsky, M. et al. A survey of the state of explainable AI for natural language processing. In Proc. 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing 447–459 (Association for Computational Linguistics, 2020).
- Li, Z. et al. Train large, then compress: rethinking model size for efficient training and inference of Transformers. In Proc. 37th International Conference on Machine Learning ICML 2020 5914–5924 (ICML, 2020).
- Shaw, D. E. et al. Anton, a special-purpose machine for molecular dynamics simulation. Commun. ACM 51, 91–97 (2008). · DOI · Paper page
- Buch, I., Giorgino, T. & De Fabritiis, G. Complete reconstruction of an enzyme-inhibitor binding process by molecular dynamics simulations. Proc. Natl Acad. Sci. USA 108, 10184–10189 (2011). · DOI · Paper page
- Ferruz, N., Harvey, M. J., Mestres, J. & De Fabritiis, G. Insights from fragment hit binding assays by molecular simulations. J. Chem. Inf. Model. 55, 2200–2205 (2015). · DOI
- Yang, K. K., Wu, Z. & Arnold, F. H. Machine-learning-guided directed evolution for protein engineering. Nat. Methods 16, 687–694 (2019). · DOI · Paper page
- Chu, S. K. S. & Siegel, J. Predicting single-point mutational effect on protein stability. In Proc. 35th Conference on Neural Information Processing Systems (NIPS, 2021).
- Hsu, C., Nisonoff, H., Fannjiang, C. & Listgarten, J. Combining evolutionary and assay-labelled data for protein fitness prediction. Preprint at https://www.biorxiv.org/content/10.1101/2021.03.28.437402v1 (2021). · DOI
- Baran, D. et al. Principles for computational design of binding antibodies. Proc. Natl Acad. Sci. USA 114, 10900–10905 (2017). · DOI · Paper page
- Dai, Z. et al. Transformer-XL: attentive language models beyond a fixed-length context. In Proc. 57th Annual Meeting for the Association for Computational Linguistics 2978–2988 (ACL, 2019). · DOI · Paper page
- Shoeybi, M. et al. Megatron-LM: training multi-billion parameter language models using model parallelism. Preprint at https://arxiv.org/abs/1909.08053 (2019). · arXiv
- Sanh, V., Debut, L., Chaumond, J. & Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. Preprint at https://arxiv.org/abs/1910.01108 (2019). · arXiv
- Gao, L. et al. The Pile: an 800-GB dataset of diverse text for language modeling. Preprint at https://arxiv.org/abs/2101.00027 (2020). · arXiv
- Clark, K., Luong, M.-T., Brain, G., Le Google Brain, Q. V. & Manning, C. D. ELECTRA: pre-training text encoders as discriminators rather than generators. Preprint at https://arxiv.org/abs/2003.10555 (2020). · arXiv
- Fedus, W., Brain, G., Zoph, B. & Shazeer, N. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res. 23, 1–39 (2022).
Cited by
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Self-citationFrom sequence to function through structure: Deep learning for protein design
(2022) · doi:10.1016/j.csbj.2022.11.014 · Cited by 124
Harnessing generative AI to decode enzyme catalysis and evolution for enhanced engineering
Xie, W. J., & Warshel, A. (2023) · National Science Review · doi:10.1093/nsr/nwad331 · Cited by 69
Rapid and automated design of two-component protein nanomaterials using ProteinMPNN
de Haas, R. J., Brunette, N., Goodson, A., Dauparas, J., Yi, S. Y., Yang, E. C., Dowling, Q., Nguyen, H., Kang, A., Bera, A. K., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314646121 · Cited by 42
Metadata
- Authors
- Ferruz, N., & Höcker, B.
- Venue
- Nature Machine Intelligence
- Published
- 2022-06-22
- Volume
- 4
- Issue
- 6
- Pages
- 521-532
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4283390570
- Cited by
- 194
- References
- 30
- FWCI
- 11.12
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Controllable protein design with language models. https://app.cochat.ai/papers/10.1038/s42256-022-00499-z
- The paper
- Ferruz, N., & Höcker, B. (2022). Controllable protein design with language models. Nature Machine Intelligence. https://doi.org/10.1038/s42256-022-00499-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.