CoChat

Structure-informed Language Models Are Protein Designers

Open access
Zheng, Zaixiang, Deng, Yifan, Xue, Dongyu, Zhou, Yi, Ye, Fei, Gu, Quanquan · 2023 · bioRxiv (Cold Spring Harbor Laboratory)
PreprintCited by 62

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.

Abstract

This paper demonstrates that language models are strong structure-based protein designers. We present LM-D esign , a generic approach to reprogramming sequence-based protein language models ( p LMs), that have learned massive sequential evolutionary knowledge from the universe of natural protein sequences, to acquire an immediate capability to design preferable protein sequences for given folds. We conduct a structural surgery on p LMs, where a lightweight structural adapter is implanted into p LMs and endows it with structural awareness. During inference, iterative refinement is performed to effectively optimize the generated protein sequences. Experiments show that LM-D esign improves the state-of-the-art results by a large margin, leading to 4% to 12% accuracy gains in sequence recovery ( e . g ., 55.65%/56.63% on CATH 4.2/4.3 single-chain benchmarks, and > 60% when designing protein complexes). We provide extensive and in-depth analyses, which verify that LM-D esign can (1) indeed leverage both structural and sequential knowledge to accurately handle structurally non-deterministic regions, (2) benefit from scaling data and model size, and (3) generalize to other proteins ( e . g ., antibodies and de novo proteins).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.02.03.526917

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 51

  1. 10.1126/science.abj8754 · doi:10.1126/science.abj8754 · Paper page
  2. Bahdanau, D. , Cho, K. , and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Confer-ence on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1409.0473. · arXiv
  3. 10.1093/nar/gkg095 · DOI · Paper page
  4. 10.1073/pnas.94.19.10172 · DOI · Paper page
  5. 2022, Controllable protein design with language models, Nature Machine Intelligence, 4, 521, 10.1038/s42256-022-00499-z · DOI · Paper page
  6. 2022, Complexity-based prompting for multi-step reasoning, arXiv preprint
  7. 2022, Pifold: Toward effec-tive and efficient protein inverse folding, arXiv preprint
  8. Gu, J. , Bradbury, J. , Xiong, C. , Li, V. O. K. , and Socher, R. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=B1l8BtlCb.
  9. 2021, Pre-training co-evolutionary protein representation via a pairwise masked language model, arXiv preprint
  10. of Proceedings of Machine Learning Research, 2022, Learning inverse folding from millions of predicted structures, Proceedings of the 39th International Conference on Ma-chine Learning, 162, 8946
  11. Hu, M. , Yuan, F. , Yang, K. K. , Ju, F. , Su, J. , Wang, H. , Yang, F. , and Ding, Q. Exploring evolution-aware &-free protein language models as protein function predictors. In Advances in Neural Information Processing Systems, 2022.
  12. Ingraham, J. , Garg, V. , Barzilay, R. , and Jaakkola, T. Gener-ative models for graph-based protein design. In Advances in neural information processing systems, 2019.
  13. Jing, B. , Eismann, S. , Suriana, P. , Townshend, R. J. L. , and Dror, R. Learning from protein structure with geomet-ric vector perceptrons. In International Conference on Learning Representations, 2020.
  14. 1983, Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features, Biopolymers: Original Research on Biomolecules, 22, 2577, 10.1002/bip.360221211 · DOI · Paper page
  15. Kingma, D. P. and Ba, J. Adam: A method for stochas-tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen-tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. · arXiv
  16. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd Interna-tional Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.org/abs/1312.6114. · arXiv
  17. 2022, Conditional antibody design as 3d equivariant graph translation, arXiv preprint
  18. 2019, What would elsa do? freezing layers during transformer fine-tuning, arXiv preprint
  19. 10.1093/nar/gkr1181 · DOI · Paper page
  20. 10.1016/s0145-305x(02)00039-3 · DOI · Paper page
  21. 2014, Direct prediction of profiles of sequences compatible with a protein structure by neural networks with fragment-based local and energy-based nonlocal profiles, Pro-teins: Structure, Function, and Bioinformatics, 82, 2565
  22. Lin, Z. , Akin, H. , Rao, R. , Hie, B. , Zhu, Z. , Lu, W. , dos Santos Costa, A. , Fazel-Zarandi, M. , Sercu, T. , Candido, S. , et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022.
  23. Lu, A. X. , Zhang, H. , Ghassemi, M. , and Moses, A. Selfsupervised contrastive learning of protein representations by mutual information maximization. BioRxiv, pp. 2020–09, 2020. · DOI · Paper page
  24. Madani, A. , Krause, B. , Greene, E. R. , Subramanian, S. , Mohr, B. P. , Holton, J. M. , Olmos Jr, J. L. , Xiong, C. , Sun, Z. Z. , Socher, R. , et al. Deep neural language modeling enables functional protein generation across families. bioRxiv, pp. 2021–07, 2021. · DOI · Paper page
  25. 2021, Adversarial contrastive pre-training for protein sequences, arXiv preprint
  26. Meier, J. , Rao, R. , Verkuil, R. , Liu, J. , Sercu, T. , and Rives, A. Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances in Neural Information Processing Systems, pp. 29287–29303, 2021. · DOI · Paper page
  27. 2022, Reprogramming large pretrained language models for antibody sequence infilling, arXiv preprint
  28. 2022, Alphafold distillation for improved inverse protein folding, arXiv preprint
  29. 2021, Pre-training of deep bidirectional protein sequence representations with structural information, IEEE Access, 9, 123912, 10.1109/ACCESS.2021.3110269 · DOI · Paper page
  30. Nambiar, A. , Heflin, M. , Liu, S. , Maslov, S. , Hopkins, M. , and Ritz, A. Transforming the language of life: transformer neural networks for protein prediction tasks. In Proceedings of the 11th ACM international conference on bioinformatics, computational biology and health informatics, pp. 1–8, 2020. · DOI · Paper page
  31. 2021, Tripletprot: deep representation learning of proteins based on siamese networks, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 19, 3744
  32. 10.1016/S0969-2126(97)00260-8 · DOI · Paper page
  33. 2020, Adapterhub: A framework for adapting transformers, arXiv preprint
  34. 2020, Mad-x: An adapter-based framework for multi-task cross-lingual transfer, arXiv preprint
  35. 10.1021/acs.jcim.0c00043 · DOI · Paper page
  36. Salazar, J. , Liang, D. , Nguyen, T. Q. , and Kirchhoff, K. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computa-tional Linguistics, pp. 2699–2712, 2020. · DOI · Paper page
  37. Savinov, N. , Chung, J. , Binkowski, M. , Elsen, E. , and van den Oord, A. Step-unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, 2021.
  38. 10.18653/v1/P16-1009 · DOI · Paper page
  39. Sohl-Dickstein, J. , Weiss, E. , Maheswaranathan, N. , and Ganguli, S. Deep unsupervised learning using nonequi-librium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
  40. 10.1016/S0969-2126(99)80062-8 · DOI
  41. 10.1093/bioinformatics/btaa003 · DOI · Paper page
  42. 2020, Profile prediction: An alignment-based pre-training task for pro-tein sequence models, arXiv preprint
  43. 2022, Generative de novo protein design with global context, arXiv preprint
  44. 2012, When a domain isn’ta domain, and why it’s important to properly filter proteins in databases: Conflicting definitions and fold classification systems for structural domains makes filtering of such databases imperative, BioEssays: news and reviews in molecular, cellular and developmental biology, 34, 1060, 10.1002/bies.201200116 · DOI
  45. 2022, Learning functional properties of proteins with language models, Nature Machine Intelligence, 4, 227, 10.1038/s42256-022-00457-9 · DOI · Paper page
  46. 2021, On memorization in probabilistic deep generative models, Advances in Neural Information Processing Systems, 34, 27916
  47. Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, L. , and Polosukhin, I. Attention is all you need. In Guyon, I. , von Luxburg, U. , Bengio, S. , Wallach, H. M. , Fergus, R. , Vishwanathan, S. V. N. , and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998–6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  48. 2021, Modeling protein using large-scale pretrain language model, arXiv preprint
  49. 2019, Machine-learningguided directed evolution for protein engineering, Nature methods, 16, 687, 10.1038/s41592-019-0496-6 · DOI · Paper page
  50. Yang, K. K. , Zanichelli, N. , and Yeh, H. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, pp. 2022–05, 2022b. · DOI · Paper page
  51. 10.1038/emboj.2011.428 · DOI

Cited by

See all 62

Metadata

Authors
Zheng, Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., & Gu, Q.
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2023-02-03
OpenAlex
W4319071809
Cited by
62
References
36
Type
posted content
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.