Structure-informed Language Models Are Protein Designers
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
This paper demonstrates that language models are strong structure-based protein designers. We present LM-D esign , a generic approach to reprogramming sequence-based protein language models ( p LMs), that have learned massive sequential evolutionary knowledge from the universe of natural protein sequences, to acquire an immediate capability to design preferable protein sequences for given folds. We conduct a structural surgery on p LMs, where a lightweight structural adapter is implanted into p LMs and endows it with structural awareness. During inference, iterative refinement is performed to effectively optimize the generated protein sequences. Experiments show that LM-D esign improves the state-of-the-art results by a large margin, leading to 4% to 12% accuracy gains in sequence recovery ( e . g ., 55.65%/56.63% on CATH 4.2/4.3 single-chain benchmarks, and > 60% when designing protein complexes). We provide extensive and in-depth analyses, which verify that LM-D esign can (1) indeed leverage both structural and sequential knowledge to accurately handle structurally non-deterministic regions, (2) benefit from scaling data and model size, and (3) generalize to other proteins ( e . g ., antibodies and de novo proteins).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 7×
“More concretely, we use conditional masked language modeling (CMLM, Ghazvininejad et al., 2019), which is proven to be very useful for conditional sequence generation in NLP (Gu et al., 2018), as the learning scheme.”
BackgroundInfluentialDenoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · cited 2×
“…2014), autoregressive models (Bahdanau et al., 2015; Vaswani et al., 2017), and diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Rombach et al., 2022), maximum likelihood estimation or its surrogates is the primary protocol for learning from massive native data…”
BackgroundBiopython: freely available python tools for computational molecular biology and bioinformatics
Cock, P. J. A., Antao, T., Chang, J. T., Chapman, B. A., Cox, C. J., Dalke, A., Friedberg, I., Hamelryck, T., Kauff, F., Wilczynski, B., et al. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp163 · Cited by 6,080 · cited 1×
“The dictionary of structural contexts is obtained by using the widely used DSSP (Kabsch & Sander, 1983) and biopython (Cock et al., 2009) tools.”
MethodAccurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,549
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · cited 2×
“Sequences were clustered at 30% sequence identity cutoff using mmseqs2 (Steinegger & S¨oding, 2017) resulting in 25,361 clusters.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Research works printed but not linked: 51
- 10.1126/science.abj8754 · doi:10.1126/science.abj8754 · Paper page
- Bahdanau, D. , Cho, K. , and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Confer-ence on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1409.0473. · arXiv
- 10.1093/nar/gkg095 · DOI · Paper page
- 10.1073/pnas.94.19.10172 · DOI · Paper page
- 2022, Controllable protein design with language models, Nature Machine Intelligence, 4, 521, 10.1038/s42256-022-00499-z · DOI · Paper page
- 2022, Complexity-based prompting for multi-step reasoning, arXiv preprint
- 2022, Pifold: Toward effec-tive and efficient protein inverse folding, arXiv preprint
- Gu, J. , Bradbury, J. , Xiong, C. , Li, V. O. K. , and Socher, R. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=B1l8BtlCb.
- 2021, Pre-training co-evolutionary protein representation via a pairwise masked language model, arXiv preprint
- of Proceedings of Machine Learning Research, 2022, Learning inverse folding from millions of predicted structures, Proceedings of the 39th International Conference on Ma-chine Learning, 162, 8946
- Hu, M. , Yuan, F. , Yang, K. K. , Ju, F. , Su, J. , Wang, H. , Yang, F. , and Ding, Q. Exploring evolution-aware &-free protein language models as protein function predictors. In Advances in Neural Information Processing Systems, 2022.
- Ingraham, J. , Garg, V. , Barzilay, R. , and Jaakkola, T. Gener-ative models for graph-based protein design. In Advances in neural information processing systems, 2019.
- Jing, B. , Eismann, S. , Suriana, P. , Townshend, R. J. L. , and Dror, R. Learning from protein structure with geomet-ric vector perceptrons. In International Conference on Learning Representations, 2020.
- 1983, Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features, Biopolymers: Original Research on Biomolecules, 22, 2577, 10.1002/bip.360221211 · DOI · Paper page
- Kingma, D. P. and Ba, J. Adam: A method for stochas-tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen-tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. · arXiv
- Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd Interna-tional Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.org/abs/1312.6114. · arXiv
- 2022, Conditional antibody design as 3d equivariant graph translation, arXiv preprint
- 2019, What would elsa do? freezing layers during transformer fine-tuning, arXiv preprint
- 10.1093/nar/gkr1181 · DOI · Paper page
- 10.1016/s0145-305x(02)00039-3 · DOI · Paper page
- 2014, Direct prediction of profiles of sequences compatible with a protein structure by neural networks with fragment-based local and energy-based nonlocal profiles, Pro-teins: Structure, Function, and Bioinformatics, 82, 2565
- Lin, Z. , Akin, H. , Rao, R. , Hie, B. , Zhu, Z. , Lu, W. , dos Santos Costa, A. , Fazel-Zarandi, M. , Sercu, T. , Candido, S. , et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022.
- Lu, A. X. , Zhang, H. , Ghassemi, M. , and Moses, A. Selfsupervised contrastive learning of protein representations by mutual information maximization. BioRxiv, pp. 2020–09, 2020. · DOI · Paper page
- Madani, A. , Krause, B. , Greene, E. R. , Subramanian, S. , Mohr, B. P. , Holton, J. M. , Olmos Jr, J. L. , Xiong, C. , Sun, Z. Z. , Socher, R. , et al. Deep neural language modeling enables functional protein generation across families. bioRxiv, pp. 2021–07, 2021. · DOI · Paper page
- 2021, Adversarial contrastive pre-training for protein sequences, arXiv preprint
- Meier, J. , Rao, R. , Verkuil, R. , Liu, J. , Sercu, T. , and Rives, A. Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances in Neural Information Processing Systems, pp. 29287–29303, 2021. · DOI · Paper page
- 2022, Reprogramming large pretrained language models for antibody sequence infilling, arXiv preprint
- 2022, Alphafold distillation for improved inverse protein folding, arXiv preprint
- 2021, Pre-training of deep bidirectional protein sequence representations with structural information, IEEE Access, 9, 123912, 10.1109/ACCESS.2021.3110269 · DOI · Paper page
- Nambiar, A. , Heflin, M. , Liu, S. , Maslov, S. , Hopkins, M. , and Ritz, A. Transforming the language of life: transformer neural networks for protein prediction tasks. In Proceedings of the 11th ACM international conference on bioinformatics, computational biology and health informatics, pp. 1–8, 2020. · DOI · Paper page
- 2021, Tripletprot: deep representation learning of proteins based on siamese networks, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 19, 3744
- 10.1016/S0969-2126(97)00260-8 · DOI · Paper page
- 2020, Adapterhub: A framework for adapting transformers, arXiv preprint
- 2020, Mad-x: An adapter-based framework for multi-task cross-lingual transfer, arXiv preprint
- 10.1021/acs.jcim.0c00043 · DOI · Paper page
- Salazar, J. , Liang, D. , Nguyen, T. Q. , and Kirchhoff, K. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computa-tional Linguistics, pp. 2699–2712, 2020. · DOI · Paper page
- Savinov, N. , Chung, J. , Binkowski, M. , Elsen, E. , and van den Oord, A. Step-unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, 2021.
- 10.18653/v1/P16-1009 · DOI · Paper page
- Sohl-Dickstein, J. , Weiss, E. , Maheswaranathan, N. , and Ganguli, S. Deep unsupervised learning using nonequi-librium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
- 10.1016/S0969-2126(99)80062-8 · DOI
- 10.1093/bioinformatics/btaa003 · DOI · Paper page
- 2020, Profile prediction: An alignment-based pre-training task for pro-tein sequence models, arXiv preprint
- 2022, Generative de novo protein design with global context, arXiv preprint
- 2012, When a domain isn’ta domain, and why it’s important to properly filter proteins in databases: Conflicting definitions and fold classification systems for structural domains makes filtering of such databases imperative, BioEssays: news and reviews in molecular, cellular and developmental biology, 34, 1060, 10.1002/bies.201200116 · DOI
- 2022, Learning functional properties of proteins with language models, Nature Machine Intelligence, 4, 227, 10.1038/s42256-022-00457-9 · DOI · Paper page
- 2021, On memorization in probabilistic deep generative models, Advances in Neural Information Processing Systems, 34, 27916
- Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, L. , and Polosukhin, I. Attention is all you need. In Guyon, I. , von Luxburg, U. , Bengio, S. , Wallach, H. M. , Fergus, R. , Vishwanathan, S. V. N. , and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998–6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
- 2021, Modeling protein using large-scale pretrain language model, arXiv preprint
- 2019, Machine-learningguided directed evolution for protein engineering, Nature methods, 16, 687, 10.1038/s41592-019-0496-6 · DOI · Paper page
- Yang, K. K. , Zanichelli, N. , and Yeh, H. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, pp. 2022–05, 2022b. · DOI · Paper page
- 10.1038/emboj.2011.428 · DOI
Cited by
Machine learning for functional protein design
Notin, P., Rollins, N., Gal, Y., Sander, C., & Marks, D. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 270
Efficient generation of protein pockets with PocketGen
Zhang, Z., Shen, W. X., Liu, Q., & Zitnik, M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00920-9 · Cited by 32
Structure-informed protein language models are robust predictors for variant effects
(2024) · doi:10.1007/s00439-024-02695-w · Cited by 30
Gamouh, H., Novotný, M., & Hoksza, D. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf431 · Cited by 23
Zhang, X., Yin, H., Ling, F., Zhan, J., & Zhou, Y. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011330 · Cited by 15
Self-citation
Metadata
- Authors
- Zheng, Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., & Gu, Q.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2023-02-03
- OpenAlex
- W4319071809
- Cited by
- 62
- References
- 36
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Structure-informed Language Models Are Protein Designers. https://app.cochat.ai/papers/10.1101/2023.02.03.526917
- The paper
- Zheng, Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., & Gu, Q. (2023). Structure-informed Language Models Are Protein Designers. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2023.02.03.526917
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.