CoChat
Sign Up
PreprintCited by 110Protein Structure and DynamicsMachine Learning in BioinformaticsComputational Drug Discovery Methods

The DOI matches its Crossref record; its reference list is not available.

Masked Inverse Folding with Sequence Transfer for Protein Representation Learning

Open access
Yang, Kevin K., Yeh, Hugh, Zanichelli, Niccolò·2022·Protein Engineering Design and Selection
DOI 10.1101/2022.05.25.493516

Preprint — published as 10.1093/protein/gzad015

Open this paper in CoChat — free accountOpen PDF

Abstract

A bstract: Self-supervised pretraining on protein sequences has led to state-of-the art performance on protein function and fitness prediction. However, sequence-only methods ignore the rich information contained in experimental and predicted protein structures. Meanwhile, inverse folding methods reconstruct a protein’s amino-acid sequence given its structure, but do not take advantage of sequences that do not have known structures. In this study, we train a masked inverse folding protein masked language model parameterized as a structured graph neural network. During pretraining, this model learns to reconstruct corrupted sequences conditioned on the backbone structure. We then show that using the outputs from a pretrained sequence-only protein masked language model as input to the inverse folding model further improves pretraining perplexity. We evaluate both of these models on downstream protein engineering tasks and analyze the effect of using information from experimental or predicted structures on performance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2022.05.25.493516

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 73 · Cited by 110

Research works printed but not linked: 71

  1. 2017, The Rosetta all-atom energy function for macromolecular modeling and design, Journal of chemical theory and computation, 13, 3031, 10.1021/acs.jctc.7b00125 · DOI · Paper page
  2. 2019, Unified rational protein engineering with sequence-based deep representation learning, Nat. Methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
  3. 2021, De novo protein design by deep network hallucination, Nature, 600, 547, 10.1038/s41586-021-04184-w · DOI · Paper page
  4. Tristan Bepler and Bonnie Berger . Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
  5. Nadav Brandes , Dan Ofer , Yam Peleg , Nadav Rappoport , and Michal Linial . ProteinBERT: A universal deep-learning model of protein sequence and function. bioRxiv, 2021. · DOI
  6. Patrick Bryant , Gabriele Pozzati , Wensi Zhu , Aditi Shenoy , Petras J. Kundrotas , and Arne Elofsson . Predicting the structure of large protein complexes using alphafold and sequential assembly. bioRxiv, 2022. · DOI · Paper page
  7. 2021, Fold2Seq: A joint sequence(1d)-fold(3d) embedding-based generative model for protein design, Proceedings of machine learning research, 139, 1261
  8. Can Chen , Jingbo Zhou , Fan Wang , Xue Liu , and Dejing Dou . Structure-aware protein self-supervised learning. ArXiv, abs/2204.04213, 2022. · DOI
  9. 10.1038/nature23912 · DOI · Paper page
  10. Ratul Chowdhury , Nazim Bouatta , Surojit Biswas , Charlotte Rochereau , George M. Church , Peter K. Sorger , and Mohammed AlQuraishi . Single-sequence protein structure prediction using language models from deep learning. bioRxiv, 2021. · DOI · Paper page
  11. 10.1126/science.278.5335.82 · DOI
  12. Christian Dallago , Jody Mou , Kadina E Johnston , Bruce Wittmann , Nick Bhattacharya , Samuel Goldman , Ali Madani , and Kevin K Yang . FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. · DOI
  13. 10.1126/science.add2187 · DOI · Paper page
  14. 10.1016/0959-440X(91)90095-B · DOI
  15. Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
  16. Ahmed Elnaggar , Michael Heinzinger , Christian Dallago , Ghalia Rehawi , Yu Wang , Llion Jones , Tom Gibbs , Tamas Feher , Christoph Angerer , Martin Steinegger , Debsindhu Bhowmik , and Burkhard Rost . ProtTrans: Towards cracking the language of life’s code through Self-Supervised learning. May 2021. · DOI
  17. 10.1021/ja058292x · DOI
  18. Fabian B. Fuchs , Daniel E. Worrall , Volker Fischer , and Max Welling . Se(3)-transformers: 3d roto-translation equivariant attention networks. ArXiv, abs/2006.10503, 2020. · arXiv
  19. 10.1126/science.282.5393.1462 · DOI
  20. 10.1126/science.abd7331 · DOI · Paper page
  21. Brian L Hie , Duo Xu , Varun R Shanker , Theodora UJ Bruun , Payton A Weidenbacher , Shaogeng Tang , and Peter S Kim . Efficient evolution of human antibodies from general protein language models and sequence information alone. bioRxiv, 2022a.
  22. 10.1016/j.cels.2022.01.003 · DOI
  23. Chloe Hsu , Robert Verkuil , Jason Liu , Zeming Lin , Brian Hie , Tom Sercu , Adam Lerer , and Alexander Rives . Learning inverse folding from millions of predicted structures. bioRxiv, 2022. · DOI
  24. John Ingraham , Vikas Garg , Regina Barzilay , and Tommi Jaakkola . Generative models for graphbased protein design. In Advances in Neural Information Processing Systems, pp. 15794–15805, 2019.
  25. Justina Jankauskaite , Brian Jiménez-García , Justas Dapkūnas , Juan Fernández-Recio , and Iain H. Moal . Skempi 2.0: An updated benchmark of changes in protein-protein binding energy, kinetics and thermodynamics upon mutation. bioRxiv, 2018. · DOI
  26. Michael Jendrusch , Jan O. Korbel , and S. Kashif Sadiq . Alphadesign: A de novo protein design framework based on alphafold. bioRxiv, 2021. · DOI · Paper page
  27. Wengong Jin , Jeremy Wohlwend , Regina Barzilay , and Tommi Jaakkola . Iterative refinement graph neural network for antibody sequence-structure co-design. arXiv preprint arXiv:2110.04624, 2021. · arXiv
  28. Bowen Jing , Stephan Eismann , Patricia Suriana , Raphael JL Townshend , and Ron Dror . Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020. · arXiv
  29. James E Lucas and Tanja Kortemme . New computational protein design methods for de novo small molecule binding sites. PLoS Computational Biology, 16, 2020. · DOI
  30. Sanaa Mansoor , Minkyung Baek , Umesh Madan , and Eric Horvitz . Toward more general embeddings for protein design: Harnessing joint representations of sequence and structure. bioRxiv, 2021. · DOI
  31. Matt McPartlon , Ben Lai , and Jinbo Xu . A deep SE (3)-equivariant model for learning inverse protein folding. bioRxiv, 2022. · DOI
  32. Joshua Meier , Roshan Rao , Robert Verkuil , Jason Liu , Tom Sercu , and Alexander Rives . Language models enable zero-shot prediction of the effects of mutations on protein function. In M. Ranzato , A. Beygelzimer , K. Nguyen , P.S. Liang , J.W. Vaughan , and Y. Dauphin (eds.), Advances in Neural Information Processing Systems 34, 2021. · DOI · Paper page
  33. Milot Mirdita , Konstantin Schütze , Yoshitaka Moriwaki , Lim Heo , Sergey Ovchinnikov , and Martin Steinegger . Colabfold - making protein folding accessible to all. 2021. · DOI · Paper page
  34. Lewis Moffat , Joe G. Greener , and David T. Jones . Using alphafold for rapid and accurate fixed backbone protein design. bioRxiv, 2021. · DOI · Paper page
  35. Lewis Moffat , Shaun M. Kandathil , and David T. Jones . Design in the dark: Learning deep generative models for de novo protein design. bioRxiv, 2022. · DOI · Paper page
  36. 10.1073/pnas.2017228118 · DOI · Paper page
  37. 2021, Structure-based protein design with deep learning, Current opinion in chemical biology, 65, 136, 10.1016/j.cbpa.2021.08.004 · DOI · Paper page
  38. Marina A Pak , Karina A Markhieva , Mariia S Novikova , Dmitry S Petrov , Ilya S Vorobyev , Ekaterina S Maksimova , Fyodor A Kondrashov , and Dmitry N Ivankov . Using alphaFold to predict the impact of single mutations on protein stability and function. BioRxiv, 2021.
  39. Alfredo Quijano-Rubio , Hsien-Wei Yeh , Jooyoung Park , Hansol Lee , Robert A. Langan , Scott E. Boyken , Marc J. Lajoie , Longxing Cao , Cameron M. Chow , Marcos C. Miranda , Jimin Wi , Hyo Jeong Hong , Lance J. Stewart , Byung-Ha Oh , and David Baker . De novo design of modular and tunable protein biosensors. Nature, 2021. · DOI
  40. Roshan Rao , Nicholas Bhattacharya , Neil Thomas , Yan Duan , Peter Chen , John Canny , Pieter Abbeel , and Yun Song . Evaluating protein transfer learning with TAPE. In Advances in Neural Information Processing Systems, pp. 9686–9698, 2019.
  41. Roshan Rao , Joshua Meier , Tom Sercu , Sergey Ovchinnikov , and Alexander Rives . Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations, 2020.
  42. 2018, Deep generative models of genetic variation capture the effects of mutations, Nature Methods, 15, 816, 10.1038/s41592-018-0138-4 · DOI · Paper page
  43. Alexander Rives , Joshua Meier , Tom Sercu , Siddharth Goyal , Zeming Lin , Jason Liu , Demi Guo , Myle Ott , C Lawrence Zitnick , Jerry Ma , and Rob Fergus . Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. U. S. A., 118(15), April 2021. · DOI · Paper page
  44. 10.1126/science.aan0693 · DOI · Paper page
  45. 10.1126/scisignal.abe1202 · DOI
  46. 10.1093/nar/gkw952 · DOI
  47. Chence Shi , Chuanrui Wang , Jiarui Lu , Bozitao Zhong , and Jian Tang . Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.08761, 2022. · arXiv
  48. 10.1126/science.1190239 · DOI
  49. 10.5522/04/7937330.v1 · DOI
  50. Vignesh Ram Somnath , Charlotte Bunne , and Andreas Krause . Multi-scale representation learning on proteins. In NeurIPS, 2021.
  51. 2020, Deep mutational scanning of sars-cov-2 receptor binding domain reveals constraints on folding and ace2 binding, Cell, 182, 1295, 10.1016/j.cell.2020.08.012 · DOI · Paper page
  52. 2018, Clustering huge protein sequence sets in linear time, Nature communications, 9, 1
  53. 2019, Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold, Nature methods, 16, 603, 10.1038/s41592-019-0437-4 · DOI · Paper page
  54. 10.1016/S0969-2126(99)80062-8 · DOI
  55. Alexey Strokach , David Becerra , Carles Corbi-Verge , Albert Perez-Riba , and Philip M Kim . Fast and flexible design of novel proteins using graph neural networks. bioRxiv, pp. 868935, 2020. · DOI
  56. 10.1093/bioinformatics/btu739 · DOI · Paper page
  57. Felix Teufel , José Juan Almagro Armenteros , Alexander Rosenberg Johansen , Magnús Halldór Gíslason , Silas Irby Pihl , Konstantinos D Tsirigos , Ole Winther , Søren Brunak , Gunnar von Heijne , and Henrik Nielsen . SignalP 6.0 achieves signal peptide prediction across all types using protein language models. bioRxiv, 2021. · DOI
  58. Vineet Thumuluri , José Juan Almagro Armenteros , Alexander Rosenberg Johansen , Henrik Nielsen , and Ole Winther . DeepLoc 2.0: multi-label subcellular localization prediction using protein language models. Nucleic Acids Research, 2022. · DOI
  59. Raphael J. L. Townshend , Rishi Bedi , Patricia Suriana , and Ron O. Dror . End-to-end learning on 3d protein structure for interface prediction. In NeurIPS, 2019.
  60. Raphael J. L. Townshend , Martin Vögele , Patricia Suriana , Alexander Derry , Alexander S Powers , Yianni Laloudakis , Sidhika Balachandar , Brandon M. Anderson , Stephan Eismann , Risi Kondor , Russ B. Altman , and Ron O. Dror . ATOM3D: Tasks on molecules in three dimensions. ArXiv, abs/2012.04035, 2021. · arXiv
  61. 10.1093/nar/gkaa1100 · DOI · Paper page
  62. Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , Łukasz Kaiser , and Illia Polosukhin . Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.
  63. 10.1002/pro.3406 · DOI
  64. Jue Wang , Sidney Lisanza , David Juergens , Doug Tischer , Ivan Anishchenko , Minkyung Baek , Joseph L Watson , Jung Ho Chun , Lukas F Milles , Justas Dauparas , et al. Deep learning methods for designing proteins scaffolding functional sites. bioRxiv, 2021. · DOI · Paper page
  65. Zichen Wang , Steven A. Combs , Ryan Brand , Miguel Calvo Rebollar , Panpan Xu , George Price , Nataliya Golovach , Emmanuel Oluwatobi Salawu , Colby Wise , Sri Priya Ponnapalli , and Peter M. Clark . Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property prediction. Scientific Reports, 12, 2022. · DOI
  66. Eli N Weinstein , Alan N Amin , Jonathan Frazer , and Debora S Marks . Non-identifiability and the blessings of misspecification in models of molecular fitness and phylogeny. bioRxiv, 2022. · DOI
  67. 10.7554/eLife.16965 · DOI
  68. Zachary Wu , S. B. Jennifer Kan , Russell D. Lewis , Bruce J. Wittmann , and Frances H. Arnold . Machine learning-assisted directed protein evolution with combinatorial libraries. Proceedings of the National Academy of Sciences USA, 2019.
  69. 10.1073/pnas.1914677117 · DOI · Paper page
  70. Kevin K Yang , Alex X Lu , and Nicolo K Fusi . Convolutions are competitive with transformers for protein sequence pretraining. bioRxiv, 2022. · DOI
  71. 10.1073/pnas.1908723117 · DOI

Metadata

Authors
Yang, K. K., Yeh, H., & Zanichelli, N.
Affiliations
  • Microsoft (United States) (US)
  • University of Chicago (US)
Corresponding author
Yang, Kevin K.
ORCID
Venue
Protein Engineering Design and Selection
Publisher
openRxiv
Published
2022-05-28
Volume
36
ISSN
0269-2139 (ISSN-L)
DOI
10.1101/2022.05.25.493516
OpenAlex
W4387966974
Cited by
110
References
73
Citations by year
2026: 21 · 2025: 30 · 2024: 42 · 2023: 15 · 2022: 2
FWCI
7.26
Type
journal article
Language
en
Open access
Locations
  • Protein Engineering Design and Selection (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, public-domain)
  • PubMed (publishedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
Keywords
protein representation learning, protein structure, protein engineering, protein function prediction, protein fitness prediction, self-supervised pre-training, sequence reconstruction
MeSH terms
Amino Acid Sequence, Protein Engineering, Protein Folding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review