Protein Design and Variant Prediction Using Autoregressive Generative Models
Open accessPreprint — published as 10.1038/s41467-021-22732-w · Paper page
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
The ability to design functional sequences and predict effects of variation is central to protein engineering and biotherapeutics. State-of-art computational methods rely on models that leverage evolutionary information but are inadequate for important applications where multiple sequence alignments are not robust. Such applications include the prediction of variant effects of indels, disordered proteins, and the design of proteins such as antibodies due to the highly variable complementarity determining regions. We introduce a deep generative model adapted from natural language processing for prediction and design of diverse functional sequences without the need for alignments. The model performs state-of-art prediction of missense and indel effects and we successfully design and test a diverse 10 5 -nanobody library that shows better expression than a 1000-fold larger synthetic library. Our results demonstrate the power of the ‘alignment-free’ autoregressive model in generalizing to regions of sequence space traditionally considered beyond the reach of prediction and design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ClinVar: improving access to variant interpretations and supporting evidence
Landrum, M. J., Lee, J. M., Benson, M., Brown, G. R., Chao, C., Chitipiralla, S., Gu, B., Hart, J., Hoffman, D., Jang, W., et al. (2018) · Nucleic Acids Research · doi:10.1093/nar/gkx1153 · Cited by 4,196
Predicting Functional Effect of Human Missense Mutations Using PolyPhen‐2
Adzhubei, I., Jordan, D. M., & Sunyaev, S. R. (2013) · Current Protocols in Human Genetics · doi:10.1002/0471142905.hg0720s76 · Cited by 3,705
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 1×
“Recently, autoregressive models have been applied across protein families to learn general features and predict mutation effects in unaligned protein sequences (Alley et al., 2019; Rives et al., 2019).”
BackgroundPan-cancer analysis of whole genomes
The ICGC/TCGA Pan-Cancer Analysis of Whole Genomes Consortium, Aaltonen, L. A., Abascal, F., Abeshouse, A., Aburatani, H., Adams, D. J., Agrawal, N., Ahn, K. S., Ahn, S.-M., Aikata, H., et al. (2020) · Nature · doi:10.1038/s41586-020-1969-6 · Cited by 3,064
Self-citationPredicting the Functional Effect of Amino Acid Substitutions and Indels
Choi, Y., Sims, G. E., Murphy, S., Miller, J. R., & Chan, A. P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046688 · Cited by 2,890
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
Nanobodies: natural single-domain antibodies
Muyldermans, S. (2013) · Annual Review of Biochemistry · doi:10.1146/annurev-biochem-063011-092449 · Cited by 2,304
BIRCH: an efficient data clustering method for very large databases
Zhang, T., Ramakrishnan, R., & Livny, M. (1996) · ACM SIGMOD Record · doi:10.1145/235968.233324 · Cited by 2,097
Generating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061 · cited 1×
“The addition of latent variables will allow controlled generation of desirable sequences (Bowman et al., 2015; Costello & Martin, 2019).”
BackgroundTacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746 · cited 1×
“Many applications in text-to-speech (Graves et al., 2013; Wang et al., 2017) and translation (Bahdanau et al., 2014; Sutskever et al., 2014) tackle context dependent word and audio prediction and generation using generative models with an autoregressive likelihood (Sutskever et al., 2011; LeCun et…”
MethodThe coming of age of de novo protein design
Huang, P.-S., Boyken, S. E., & Baker, D. (2016) · Nature · doi:10.1038/nature19946 · Cited by 1,478
Deep mutational scanning: a new style of protein science
Fowler, D. M., & Fields, S. (2014) · Nature Methods · doi:10.1038/nmeth.3027 · Cited by 1,287 · cited 1×
“High-throughput techniques have opened up opportunities to phenotype thousands of variants in a single experiment (Kosuri & Church, 2014; Fowler & Fields, 2014), but the sampled diversity of sequences pales in comparison to that found in nature.”
BackgroundProtein 3D Structure Computed from Evolutionary Sequence Variation
Marks, D. S., Colwell, L. J., Sheridan, R., Hopf, T. A., Pagnani, A., Zecchina, R., & Sander, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028766 · Cited by 1,144
Self-citationUnified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Clustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875 · cited 1×
“Due to the large number of sequences in the llama immune repertoire, sequence weights were approximated using Linclust (Steinegger & S¨oding, 2018) by clustering sequences at both 80% and 90% sequence identity thresholds.”
MethodMutation effects predicted from sequence co-variation
Hopf, T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. P. I., Springer, M., Sander, C., & Marks, D. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3769 · Cited by 775
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719
Biophysical properties of the clinical-stage antibody landscape
Jain, T., Sun, T., Durand, S., Hall, A., Houston, N. R., Nett, J. H., Sharkey, B., Bobrowicz, B., Caffry, I., Yu, Y., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1616408114 · Cited by 629
Ovchinnikov, S., Kamisetty, H., & Baker, D. (2014) · eLife · doi:10.7554/elife.02030 · Cited by 629
Protein structure prediction from sequence variation
Marks, D. S., Hopf, T. A., & Sander, C. (2012) · Nature Biotechnology · doi:10.1038/nbt.2419 · Cited by 622
Self-citationDeep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601
Self-citationSequence co-evolution gives 3D contacts and structures of protein complexes
Hopf, T. A., Schärfe, C. P. I., Rodrigues, J. P. G. L. M., Green, A. G., Kohlbacher, O., Sander, C., Bonvin, A. M. J. J., & Marks, D. S. (2014) · eLife · doi:10.7554/elife.03430 · Cited by 515
Self-citationNeural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · cited 1×
“…we parameterize this process with dilated convolutional neural networks (Figure 1b), which are feed-forward deep neural networks that aggregate long-range dependencies in sequences over an exponentially-large receptive field (van den Oord et al., 2016; Kalchbrenner et al., 2016; Gupta & Rush, 2017).”
MethodHigh-throughput sequencing of the paired human immunoglobulin heavy and light chain repertoire
DeKosky, B. J., Ippolito, G. C., Deschner, R. P., Lavinder, J. J., Wine, Y., Rawlings, B. M., Varadarajan, N., Giesecke, C., Dörner, T., Andrews, S. F., et al. (2013) · Nature Biotechnology · doi:10.1038/nbt.2492 · Cited by 410
Grimm, D. G., Azencott, C., Aicheler, F., Gieraths, U., MacArthur, D. G., Samocha, K. E., Cooper, D. N., Stenson, P. D., Daly, M. J., Smoller, J. W., et al. (2015) · Human Mutation · doi:10.1002/humu.22768 · Cited by 356
Protein interaction networks revealed by proteome coevolution
Cong, Q., Anishchenko, I., Ovchinnikov, S., & Baker, D. (2019) · Science · doi:10.1126/science.aaw6718 · Cited by 241
Antibody complementarity determining region design using high-capacity machine learning
Liu, G., Zeng, H., Mueller, J., Carter, B., Wang, Z., Schilz, J., Horny, G., Birnbaum, M. E., Ewert, S., & Gifford, D. K. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz895 · Cited by 157
Large-scale network analysis reveals the sequence space architecture of antibody repertoires
Miho, E., Roškar, R., Greiff, V., & Reddy, S. T. (2019) · Nature Communications · doi:10.1038/s41467-019-09278-8 · Cited by 138
Green, A. G., Elhabashy, H., Brock, K. P., Maddamsetti, R., Kohlbacher, O., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-21636-z · Cited by 114
Self-citationLinder, J., Bogard, N., Rosenberg, A. B., & Seelig, G. (2020) · Cell Systems · doi:10.1016/j.cels.2020.05.007 · Cited by 107
Research works printed but not linked: 76
- 10.1038/nrm2805 · DOI · Paper page
- 10.1016/j.copbio.2009.08.005 · DOI · Paper page
- 10.1002/pro.481 · DOI · Paper page
- 10.1093/protein/gzw042 · DOI
- 10.1038/s41594-018-0028-6 · DOI · Paper page
- 10.1038/nbt.1791 · DOI · Paper page
- Schoof, M. et al. An ultra-potent synthetic nanobody neutralizes SARS-CoV-2 by locking Spike into an inactive conformation. bioRxiv, 2020.2008.2008.238469 (2020).
- 10.1074/jbc.REV120.010181 · DOI · Paper page
- 2016, Maximum-Entropy Models of Sequenced Immune Repertoires Predict Antigen-Antibody Affinity, PLoS Comput Biol, 12, e1004870, 10.1371/journal.pcbi.1004870 · DOI
- 10.1073/pnas.1001705107 · DOI · Paper page
- 10.1038/s41467-018-02832-w · DOI · Paper page
- 10.1073/pnas.1525510113 · DOI · Paper page
- Muyldermans, S. A guide to: generation and design of nanobodies. FEBS J (2020). · DOI · Paper page
- Zimmermann, I. et al. Synthetic single domain antibodies for the conformational trapping of membrane proteins. Elife 7 (2018). · DOI · Paper page
- 10.1016/j.cell.2012.04.012 · DOI · Paper page
- 10.1073/pnas.1314045110 · DOI · Paper page
- Green, A.G. et al. Proteome-scale discovery of protein interactions with residue-level resolution using sequence coevolution. bioRxiv, 791293 (2019). · DOI
- 2018, Quantitative Missense Variant Effect Prediction Using Large-Scale Mutagenesis Data, Cell Syst, 6, 116, 10.1016/j.cels.2017.11.003 · DOI · Paper page
- 10.1371/journal.pcbi.1003776 · DOI · Paper page
- 2015, Coevolutionary Landscape Inference and the Context-Dependence of Mutations in Beta-Lactamase TEM-(I), Molecular Biology and Evolution, 33, 268
- 2020, Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations, Mol Syst Biol, 16, e9380, 10.15252/msb.20199380 · DOI · Paper page
- 10.1093/nar/gks539 · DOI · Paper page
- 10.1038/ng.2892 · DOI · Paper page
- 10.1093/nar/gkr407 · DOI · Paper page
- 10.1016/S0145-305X(02)00039-3 · DOI · Paper page
- 10.1021/cr400525m · DOI · Paper page
- 10.1093/hmg/ddq400 · DOI · Paper page
- 2017, Effects of short indels on protein structure and function in human genomes, Sci Rep, 7, 9313, 10.1038/s41598-017-09287-x · DOI · Paper page
- 10.1101/gr.115907.110 · DOI · Paper page
- 10.1002/prot.22504 · DOI · Paper page
- 10.1016/S1470-2045(17)30516-8 · DOI · Paper page
- 2019, Intrinsically disordered proteins and structured proteins with intrinsically disordered regions have different functional roles in the cell, PLoS One, 14, e0217889, 10.1371/journal.pone.0217889 · DOI · Paper page
- 10.1186/1471-2164-10-S3-S7 · DOI
- Graves, A. , Mohamed, A. & Hinton, G. Speech recognition with deep recurrent neural networks. 2013 IEEE International Conference on Acoustics, Speech and Singal Processing, 6645–6649 (2013). · DOI · Paper page
- Bahdanau, D. , Cho, K. & Begio, Y. Neural machine translation by jointly learning to align and translate. arXiv (2014).
- Sutskever, I. , Vinyals, O. & Le, Q.V. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 3104–3112 (2014).
- 10.1093/bioinformatics/btaa003 · DOI · Paper page
- 2019, Conditioning by adaptive sampling for robust design, Proceedings of the 36th International Conference on Machine Learning, 97, 773
- 10.1371/journal.ppat.1004552 · DOI
- 2017, Naive Human Antibody Libraries for Infectious Diseases, Adv Exp Med Biol, 1053, 35, 10.1007/978-3-319-72077-7_3 · DOI
- 2018, Scalable, Continuous Evolution of Genes at Mutation Rates above Genomic Error Thresholds, Cell, 175, 1946, 10.1016/j.cell.2018.10.021 · DOI · Paper page
- Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems, 5998–6008 (2017).
- Sutskever, I. , Martens, J. & Hinton, G. Generating text with recurrent neural networks. Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1017–1024 (2011).
- Pascanu, R. , Mikolov, T. & Begio, Y. On the difficulty of training recurrent neural networks. International Conference on Machine Learning, 1310-1318 (2013).
- Cho, K. et al. Learning phrase representations using RNN Encoder-Decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1724–1734 (2014). · DOI · Paper page
- van den Oord, A. et al. Wavenet: A generative model for raw audio. arXiv (2016).
- Gupta, A. & Rush, A. Dilated convolutions for modeling long-distance genomic dependencies. arXiv (2017). · DOI · Paper page
- 2017, Exploiting sequence and stability information for directing nanobody stability engineering, Biochim Biophys Acta Gen Subj, 1861, 2196, 10.1016/j.bbagen.2017.06.014 · DOI
- 2017, Evaluation of anti-botulinum neurotoxin single domain antibodies with additional optimization for improved production and stability, Toxicon, 135, 51, 10.1016/j.toxicon.2017.06.002 · DOI
- 2015, Improving the biophysical properties of anti-ricin single-domain antibodies, Biotechnol Rep (Amst), 6, 27, 10.1016/j.btre.2015.01.001 · DOI
- 10.1371/journal.pone.0077678 · DOI
- Durbin, R. , Eddy, S.R. , Krogh, A. & Mitchison, G. Biological sequence analysis: probabilistic models of proteins and nucleic acids. (Cambridge university press, 1998). · DOI · Paper page
- 2011, Accelerated profile HMM searches, PLoS computational biology, 7, e1002195, 10.1371/journal.pcbi.1002195 · DOI · Paper page
- 10.1016/j.ajhg.2018.03.018 · DOI · Paper page
- 2019, An experimental assay of the interactions of amino acids from orthologous sequences shaping a complex fitness landscape, PLoS Genet, 15, e1008079, 10.1371/journal.pgen.1008079 · DOI · Paper page
- 10.1016/S0161-5890(00)00081-X · DOI · Paper page
- 10.4161/mabs.12187 · DOI
- 2012, Antibody polyspecificity: what does it matter?, Adv Exp Med Biol, 750, 213, 10.1007/978-1-4614-3461-0_16 · DOI
- 10.4049/jimmunol.1300880 · DOI
- 10.1080/19420862.2017.1356528 · DOI
- Lim, C.C. , Choong, Y.S. & Lim, T.S. Cognizance of Molecular Methods for the Generation of Mutagenic Phage Display Antibody Libraries for Affinity Maturation. Int J Mol Sci 20 (2019). · DOI
- 2017, Antibody repertoire profiling with mimotope arrays, Hum Vaccin Immunother, 13, 314, 10.1080/21645515.2017.1264786 · DOI
- 10.1126/science.1086907 · DOI · Paper page
- Wellner, A. et al. Rapid generation of potent antibodies by autonomous hypermutation in yeast. bioRxiv (2020). · doi:10.1101/2020.11.11.378778
- Ingraham, J.B. , Vikas, G.K. , Barzilay, R. & Jaakkola, T. Generative models for graph-based protein design. 33rd Conference on Neural Information Processing Systems 15794–15805 (2019).
- Kim, Y. , Wiseman, S. , Miller, A.C. , Sontag, D. & Rush, A. Semi-amortized variational autoencoders. arXiv (2018).
- Yang, Z. , Hu, Z. , Salakhutdinov, R. & Berg-Kirkpatrick, T. Improved variational autoencoders for text modeling using dilated convolutions. arXiv (2017).
- 2018, Design of metalloproteins and novel protein folds using variational autoencoders, Sci Rep, 8, 16189, 10.1038/s41598-018-34533-1 · DOI · Paper page
- Vijayakumar, A.K. et al. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv (2016).
- Salimans, T. & Kingma, D.P. Weight normalization: a simple reparametrization to accelerate training of deep neural networks. Advances in neural information processing systems, 901–909 (2016).
- Ba, J.L. , Kiros, J.R. & Hinton, G. Layer normalization. arXiv (2016).
- 2014, Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 15, 1929
- Kingma, D.P. & Ba, J.L. Adam: a method for stochastic optimization. arXiv (2014).
- 10.1016/0022-2836(82)90515-0 · DOI · Paper page
- 10.1002/elps.11501401163 · DOI · Paper page
- 10.1093/bioinformatics/btp163 · DOI · Paper page
Cited by
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635
Low-N protein engineering with data-efficient deep learning
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., & Church, G. M. (2021) · Nature Methods · doi:10.1038/s41592-021-01100-y · Cited by 422
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Expanding functional protein sequence spaces using generative adversarial networks
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00310-5 · Cited by 388
Metadata
- Authors
- Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-09-05
- OpenAlex
- W2971803505
- Cited by
- 32
- References
- 33
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.