Protein Design and Variant Prediction Using Autoregressive Generative Models
Open accessPreprint — published as 10.1038/s41467-021-22732-w · Paper page
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
The ability to design functional sequences and predict effects of variation is central to protein engineering and biotherapeutics. State-of-art computational methods rely on models that leverage evolutionary information but are inadequate for important applications where multiple sequence alignments are not robust. Such applications include the prediction of variant effects of indels, disordered proteins, and the design of proteins such as antibodies due to the highly variable complementarity determining regions. We introduce a deep generative model adapted from natural language processing for prediction and design of diverse functional sequences without the need for alignments. The model performs state-of-art prediction of missense and indel effects and we successfully design and test a diverse 10 5 -nanobody library that shows better expression than a 1000-fold larger synthetic library. Our results demonstrate the power of the ‘alignment-free’ autoregressive model in generalizing to regions of sequence space traditionally considered beyond the reach of prediction and design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ClinVar: improving access to variant interpretations and supporting evidence
Landrum, M. J., Lee, J. M., Benson, M., Brown, G. R., Chao, C., Chitipiralla, S., Gu, B., Hart, J., Hoffman, D., Jang, W., et al. (2018) · Nucleic Acids Research · doi:10.1093/nar/gkx1153 · Cited by 4,196
Predicting Functional Effect of Human Missense Mutations Using PolyPhen‐2
Adzhubei, I., Jordan, D. M., & Sunyaev, S. R. (2013) · Current Protocols in Human Genetics · doi:10.1002/0471142905.hg0720s76 · Cited by 3,705
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 1×
“Recently, autoregressive models have been applied across protein families to learn general features and predict mutation effects in unaligned protein sequences (Alley et al., 2019; Rives et al., 2019).”
BackgroundPan-cancer analysis of whole genomes
The ICGC/TCGA Pan-Cancer Analysis of Whole Genomes Consortium, Aaltonen, L. A., Abascal, F., Abeshouse, A., Aburatani, H., Adams, D. J., Agrawal, N., Ahn, K. S., Ahn, S.-M., Aikata, H., et al. (2020) · Nature · doi:10.1038/s41586-020-1969-6 · Cited by 3,064
Self-citationPredicting the Functional Effect of Amino Acid Substitutions and Indels
Choi, Y., Sims, G. E., Murphy, S., Miller, J. R., & Chan, A. P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046688 · Cited by 2,890
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
Nanobodies: natural single-domain antibodies
Muyldermans, S. (2013) · Annual Review of Biochemistry · doi:10.1146/annurev-biochem-063011-092449 · Cited by 2,304
Research works printed but not linked: 76
- 10.1038/nrm2805 · DOI · Paper page
- 10.1016/j.copbio.2009.08.005 · DOI · Paper page
- 10.1002/pro.481 · DOI · Paper page
- 10.1093/protein/gzw042 · DOI
- 10.1038/s41594-018-0028-6 · DOI · Paper page
- 10.1038/nbt.1791 · DOI · Paper page
- Schoof, M. et al. An ultra-potent synthetic nanobody neutralizes SARS-CoV-2 by locking Spike into an inactive conformation. bioRxiv, 2020.2008.2008.238469 (2020).
- 10.1074/jbc.REV120.010181 · DOI · Paper page
- 2016, Maximum-Entropy Models of Sequenced Immune Repertoires Predict Antigen-Antibody Affinity, PLoS Comput Biol, 12, e1004870, 10.1371/journal.pcbi.1004870 · DOI
- 10.1073/pnas.1001705107 · DOI · Paper page
- 10.1038/s41467-018-02832-w · DOI · Paper page
- 10.1073/pnas.1525510113 · DOI · Paper page
- Muyldermans, S. A guide to: generation and design of nanobodies. FEBS J (2020). · DOI · Paper page
- Zimmermann, I. et al. Synthetic single domain antibodies for the conformational trapping of membrane proteins. Elife 7 (2018). · DOI · Paper page
- 10.1016/j.cell.2012.04.012 · DOI · Paper page
- 10.1073/pnas.1314045110 · DOI · Paper page
- Green, A.G. et al. Proteome-scale discovery of protein interactions with residue-level resolution using sequence coevolution. bioRxiv, 791293 (2019). · DOI
- 2018, Quantitative Missense Variant Effect Prediction Using Large-Scale Mutagenesis Data, Cell Syst, 6, 116, 10.1016/j.cels.2017.11.003 · DOI · Paper page
- 10.1371/journal.pcbi.1003776 · DOI · Paper page
- 2015, Coevolutionary Landscape Inference and the Context-Dependence of Mutations in Beta-Lactamase TEM-(I), Molecular Biology and Evolution, 33, 268
- 2020, Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations, Mol Syst Biol, 16, e9380, 10.15252/msb.20199380 · DOI · Paper page
- 10.1093/nar/gks539 · DOI · Paper page
- 10.1038/ng.2892 · DOI · Paper page
- 10.1093/nar/gkr407 · DOI · Paper page
- 10.1016/S0145-305X(02)00039-3 · DOI · Paper page
- 10.1021/cr400525m · DOI · Paper page
- 10.1093/hmg/ddq400 · DOI · Paper page
- 2017, Effects of short indels on protein structure and function in human genomes, Sci Rep, 7, 9313, 10.1038/s41598-017-09287-x · DOI · Paper page
- 10.1101/gr.115907.110 · DOI · Paper page
- 10.1002/prot.22504 · DOI · Paper page
- 10.1016/S1470-2045(17)30516-8 · DOI · Paper page
- 2019, Intrinsically disordered proteins and structured proteins with intrinsically disordered regions have different functional roles in the cell, PLoS One, 14, e0217889, 10.1371/journal.pone.0217889 · DOI · Paper page
- 10.1186/1471-2164-10-S3-S7 · DOI
- Graves, A. , Mohamed, A. & Hinton, G. Speech recognition with deep recurrent neural networks. 2013 IEEE International Conference on Acoustics, Speech and Singal Processing, 6645–6649 (2013). · DOI · Paper page
- Bahdanau, D. , Cho, K. & Begio, Y. Neural machine translation by jointly learning to align and translate. arXiv (2014).
- Sutskever, I. , Vinyals, O. & Le, Q.V. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 3104–3112 (2014).
- 10.1093/bioinformatics/btaa003 · DOI · Paper page
- 2019, Conditioning by adaptive sampling for robust design, Proceedings of the 36th International Conference on Machine Learning, 97, 773
- 10.1371/journal.ppat.1004552 · DOI
- 2017, Naive Human Antibody Libraries for Infectious Diseases, Adv Exp Med Biol, 1053, 35, 10.1007/978-3-319-72077-7_3 · DOI
- 2018, Scalable, Continuous Evolution of Genes at Mutation Rates above Genomic Error Thresholds, Cell, 175, 1946, 10.1016/j.cell.2018.10.021 · DOI · Paper page
- Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems, 5998–6008 (2017).
- Sutskever, I. , Martens, J. & Hinton, G. Generating text with recurrent neural networks. Proceedings of the 28th International Conference on Machine Learning (ICML-11), 1017–1024 (2011).
- Pascanu, R. , Mikolov, T. & Begio, Y. On the difficulty of training recurrent neural networks. International Conference on Machine Learning, 1310-1318 (2013).
- Cho, K. et al. Learning phrase representations using RNN Encoder-Decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1724–1734 (2014). · DOI · Paper page
- van den Oord, A. et al. Wavenet: A generative model for raw audio. arXiv (2016).
- Gupta, A. & Rush, A. Dilated convolutions for modeling long-distance genomic dependencies. arXiv (2017). · DOI · Paper page
- 2017, Exploiting sequence and stability information for directing nanobody stability engineering, Biochim Biophys Acta Gen Subj, 1861, 2196, 10.1016/j.bbagen.2017.06.014 · DOI
- 2017, Evaluation of anti-botulinum neurotoxin single domain antibodies with additional optimization for improved production and stability, Toxicon, 135, 51, 10.1016/j.toxicon.2017.06.002 · DOI
- 2015, Improving the biophysical properties of anti-ricin single-domain antibodies, Biotechnol Rep (Amst), 6, 27, 10.1016/j.btre.2015.01.001 · DOI
- 10.1371/journal.pone.0077678 · DOI
- Durbin, R. , Eddy, S.R. , Krogh, A. & Mitchison, G. Biological sequence analysis: probabilistic models of proteins and nucleic acids. (Cambridge university press, 1998). · DOI · Paper page
- 2011, Accelerated profile HMM searches, PLoS computational biology, 7, e1002195, 10.1371/journal.pcbi.1002195 · DOI · Paper page
- 10.1016/j.ajhg.2018.03.018 · DOI · Paper page
- 2019, An experimental assay of the interactions of amino acids from orthologous sequences shaping a complex fitness landscape, PLoS Genet, 15, e1008079, 10.1371/journal.pgen.1008079 · DOI · Paper page
- 10.1016/S0161-5890(00)00081-X · DOI · Paper page
- 10.4161/mabs.12187 · DOI
- 2012, Antibody polyspecificity: what does it matter?, Adv Exp Med Biol, 750, 213, 10.1007/978-1-4614-3461-0_16 · DOI
- 10.4049/jimmunol.1300880 · DOI
- 10.1080/19420862.2017.1356528 · DOI
- Lim, C.C. , Choong, Y.S. & Lim, T.S. Cognizance of Molecular Methods for the Generation of Mutagenic Phage Display Antibody Libraries for Affinity Maturation. Int J Mol Sci 20 (2019). · DOI
- 2017, Antibody repertoire profiling with mimotope arrays, Hum Vaccin Immunother, 13, 314, 10.1080/21645515.2017.1264786 · DOI
- 10.1126/science.1086907 · DOI · Paper page
- Wellner, A. et al. Rapid generation of potent antibodies by autonomous hypermutation in yeast. bioRxiv (2020). · doi:10.1101/2020.11.11.378778
- Ingraham, J.B. , Vikas, G.K. , Barzilay, R. & Jaakkola, T. Generative models for graph-based protein design. 33rd Conference on Neural Information Processing Systems 15794–15805 (2019).
- Kim, Y. , Wiseman, S. , Miller, A.C. , Sontag, D. & Rush, A. Semi-amortized variational autoencoders. arXiv (2018).
- Yang, Z. , Hu, Z. , Salakhutdinov, R. & Berg-Kirkpatrick, T. Improved variational autoencoders for text modeling using dilated convolutions. arXiv (2017).
- 2018, Design of metalloproteins and novel protein folds using variational autoencoders, Sci Rep, 8, 16189, 10.1038/s41598-018-34533-1 · DOI · Paper page
- Vijayakumar, A.K. et al. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv (2016).
- Salimans, T. & Kingma, D.P. Weight normalization: a simple reparametrization to accelerate training of deep neural networks. Advances in neural information processing systems, 901–909 (2016).
- Ba, J.L. , Kiros, J.R. & Hinton, G. Layer normalization. arXiv (2016).
- 2014, Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 15, 1929
- Kingma, D.P. & Ba, J.L. Adam: a method for stochastic optimization. arXiv (2014).
- 10.1016/0022-2836(82)90515-0 · DOI · Paper page
- 10.1002/elps.11501401163 · DOI · Paper page
- 10.1093/bioinformatics/btp163 · DOI · Paper page
Cited by
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635
Low-N protein engineering with data-efficient deep learning
Biswas, S., Khimulya, G., Alley, E. C., Esvelt, K. M., & Church, G. M. (2021) · Nature Methods · doi:10.1038/s41592-021-01100-y · Cited by 422
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Expanding functional protein sequence spaces using generative adversarial networks
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00310-5 · Cited by 388
Metadata
- Authors
- Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-09-05
- OpenAlex
- W2971803505
- Cited by
- 32
- References
- 33
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Protein Design and Variant Prediction Using Autoregressive Generative Models. https://app.cochat.ai/papers/10.1101/757252
- The paper
- Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2019). Protein Design and Variant Prediction Using Autoregressive Generative Models. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/757252
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.