CoChat
Sign Up
Cited by 107RNA and protein synthesis mechanismsCRISPR and Genetic EngineeringMachine Learning in Bioinformatics

A Generative Neural Network for Maximizing Fitness and Diversity of Synthetic DNA and Protein Sequences

Open access
Linder, Johannes, Bogard, Nicholas, Rosenberg, Alexander B., Seelig, Georg·2020·Cell Systems
DOI 10.1016/j.cels.2020.05.007PMC PMC8694568
Open this paper in CoChat — free accountOpen PDF

Abstract

Engineering gene and protein sequences with defined functional properties is a major goal of synthetic biology. Deep neural network models, together with gradient ascent-style optimization, show promise for sequence design. The generated sequences can however get stuck in local minima and often have low diversity. Here, we develop deep exploration networks (DENs), a class of activation-maximizing generative models, which minimize the cost of a neural network fitness predictor by gradient descent. By penalizing any two generated patterns on the basis of a similarity metric, DENs explicitly maximize sequence diversity. To avoid drifting into low-confidence regions of the predictor, we incorporate variational autoencoders to maintain the likelihood ratio of generated sequences. Using DENs, we engineered polyadenylation signals with more than 10-fold higher selection odds than the best gradient ascent-generated patterns, identified splice regulatory sequences predicted to result in highly differential splicing between cell lines, and improved on state-of-the-art results for protein design tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.cels.2020.05.007

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Enformer can make predictions for arbitrary sequences, which enables the synthetic design of enhancers that are optimized to exhibit cell-type specificity.

    “it can make predictions for arbitrary sequences, which enables the synthetic design of enhancers that are optimized to exhibit cell-type specificity31”

    From Effective gene expression prediction from sequence by integrating long-range interactions (2021) · cites this paper as [31]

    The Discussion (page unavailable) proposes that DENs could be used with gene-expression data to engineer cell-type-specific enhancers. The paper does not attribute arbitrary-sequence prediction or this design capability to Enformer, and presents cell-type-specific enhancer design as a possibility rather than a demonstrated result. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G, Davis A, Dean J, Devin M, (2016). Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv, arXiv:1603.04467. · arXiv
  2. Alipanahi B, Delong A, Weirauch MT, and Frey BJ (2015). Predicting the sequence specificities of dnaand rna-binding proteins by deep learning. Nat. Biotechnol 33, 831– 838. 26213851
  3. Ardizzone L, Kruse J, Wirkert S, Rahner D, Pellegrini E, Klessen R, Maier-Hein L, Rother C, and Köthe U (2018). Analyzing inverse problems with invertible neural networks. arXiv, arXiv:1808.04730. · arXiv
  4. Avsec Z, Weilert M, Shrikumar A, Alexandari A, Krueger S, Dalal K, Fropf R, McAnany C, Gagneur J, Kundaje A, and Zeitlinger J (2019). Deep learning at base-resolution reveals motif syntax of the cis-regulatory code. bioRxiv. 10.1101/737981. PMC8812996 33603233 · DOI
  5. Bastien F, Lamblin P, Pascanu R, Bergstra J, Goodfellow I, Bergeron A, Bouchard N, Warde-Farley D, and Bengio Y (2012). Theano: new features and speed improvements. arXiv, arXiv:1211.5590. · arXiv
  6. Bengio Y, Léonard N, and Courville A (2013). Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv, arXiv:1308.3432. · arXiv
  7. Biswas S, Kuznetsov G, Ogden P, Conway N, Adams R, and Church G (2018). Toward machine-guided design of proteins. bioRxiv. 10.1101/337154. · DOI
  8. Blei DM, Kucukelbir A, and McAuliffe JD (2017). Variational inference: a review for statisticians. J. Am. Stat. Assoc 112, 859– 877.
  9. Blencowe BJ (2006). Alternative splicing: new insights from global analyses. Cell 126, 37– 47. 16839875
  10. Bogard N, Linder J, Rosenberg AB, and Seelig G (2019). A deep neural network for predicting and engineering alternative polyadenylation. Cell 178, 91– 106.e23. 31178116 PMC6599575
  11. Brookes D, Park H, and Listgarten J (2019). Conditioning by adaptive sampling for robust design. arXiv, arXiv:1901.10060. · arXiv
  12. Chen Z, Boyken SE, Jia M, Busch F, Flores-Solis D, Bick MJ, Lu P, VanAernum ZL, Sahasrabuddhe A, Langan RA, (2019). Programmable design of orthogonal protein heterodimers. Nature 565, 106– 111. 30568301 PMC6537907
  13. Chuai G, Ma H, Yan J, Chen M, Hong N, Xue D, Zhou C, Zhu C, Chen K, Duan B, (2018). Deepcrispr: optimized Crispr guide rna design by deep learning. Genome Biol. 19, 80. 29945655 PMC6020378
  14. Costello Z, and Martin H (2019). How to hallucinate functional proteins. arXiv, arXiv:1903.00458. · arXiv
  15. Courbariaux M, Hubara I, Soudry D, El-Yaniv R, and Bengio Y (2016). Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or −1. arXiv, arXiv:1602.02830. · arXiv
  16. Cuperus JT, Groves B, Kuchina A, Rosenberg AB, Jojic N, Fields S, and Seelig G (2017). Deep learning of the regulatory grammar of yeast 5′ un-translated regions from 500,000 random sequences. Genome Res. 27, 2015– 2024. 29097404 PMC5741052
  17. Di Giammartino DC, Nishida K, and Manley JL (2011). Mechanisms and consequences of alternative polyadenylation. Mol. Cell 43, 853– 866. 21925375 PMC3194005
  18. Eiben AE, and Smith J (2015). From evolutionary computation to the evolution of things. Nature 521, 476– 482. 26017447
  19. Elkon R, Ugalde AP, and Agami R (2013). Alternative cleavage and polyadenylation: extent, regulation and function. Nat. Rev. Genet 14, 496– 506. 23774734
  20. Eraslan G, Avsec Ž, Gagneur J, and Theis FJ (2019). Deep learning: new computational modelling techniques for genomics. Nat. Rev. Genet 20, 389– 403. 30971806
  21. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, and Bengio Y (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems 27, Ghahramani Z, Welling M, Cortes C, Lawrence ND, and Weinberger KQ, eds. (Curran Associates, Inc.), pp. 2672– 2680.
  22. Greenside P, Shimko T, Fordyce P, and Kundaje A (2018). Discovering epistatic feature interactions from neural network models of regulatory dna sequences. Bioinformatics 34, i629– i637. 30423062 PMC6129272
  23. Jaganathan K, Panagiotopoulou SK, McRae JF, Darbandi SF, Knowles D, Li YI, Kosmicki JA, Arbelaez J, Cui W, Schwartz GB, (2019). Predicting splicing from primary sequence with deep learning. Cell 176, 535– 548. 30661751
  24. Jang E, Gu S, and Poole B (2016). Categorical reparameterization with gumbel-softmax. arXiv, arXiv:1611.01144. · arXiv
  25. Kelley DR, Reshef YA, Bileschi M, Belanger D, McLean CY, and Snoek J (2018). Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Res. 28, 739– 750. 29588361 PMC5932613
  26. Kelley DR, Snoek J, and Rinn JL (2016). Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res. 26, 990– 999. 27197224 PMC4937568
  27. Killoran N, Lee L, Delong A, Duvenaud D, and Frey B (2017). Generating and designing dna with deep generative models. arXiv, arXiv:1712.06148. · arXiv
  28. Kingma D, and Welling M (2013). Auto-encoding variational bayes. arXiv, arXiv:1312.6114. · arXiv
  29. Kirkpatrick S, Gelatt CD, and Vecchi MP (1983). Optimization by simulated annealing. Science 220, 671– 680. 17813860
  30. Lakshminarayanan B, Pritzel A, and Blundell C (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6402– 6413.
  31. Lanchantin J, Singh R, Lin Z, and Qi Y (2016). Deep motif: visualizing genomic sequence classifications. arXiv, arXiv:1605.01133. · arXiv
  32. Lee Y, and Rio DC (2015). Mechanisms and regulation of alternative pre-mRNA splicing. Annu. Rev. Biochem 84, 291– 323. 25784052 PMC4526142
  33. Metropolis N, Rosenbluth AW, Rosenbluth MN, Teller AH, and Teller E (1953). Equation of state calculations by fast computing machines. J. Chem. Phys 21, 1087– 1092. 41342507
  34. Mirjalili S, Dong J, Sadiq A, and Faris H (2020). Genetic algorithm: theory, literature review, and application in image reconstruction. In Nature-Inspired Optimizers. Studies in Computational Intelligence, Mirjalili S, Song Dong J, and Lewis A, eds. (Springer), pp. 69– 85.
  35. Mirza M, and Osindero S (2014). Conditional generative adversarial nets. arXiv, arXiv:1411.1784. · arXiv
  36. Movva R, Greenside P, Marinov GK, Nair S, Shrikumar A, and Kundaje A (2019). Deciphering regulatory dna sequences and noncoding genetic variants using neural network models of massively parallel reporter assays. PLoS One 14, e0218073. 31206543 PMC6576758
  37. Quang D, and Xie X (2019). Factornet: a deep learning framework for predicting cell type specific transcription factor binding from nucleotide-resolution sequential data. Methods 166, 40– 47. 30922998 PMC6708499
  38. Radford A, Metz L, and Chintala S (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv, arXiv:1511.06434. · arXiv
  39. Repecka D, Jauniskis V, Karpus L, Rembeza E, Zrimec J, Poviloniene S, Rokaitis I, Laurynenas A, Abuajwa W, Savolainen O, (2019). Expanding functional protein sequence space using generative adversarial networks. bioRxiv. 10.1101/789719. · DOI
  40. Riesselman A, Shin J, Kollasch A, McMahon C, Simon E, Sander C, Manglik A, Kruse A, and Marks D (2019). Accelerating protein design using autoregressive generative models. bioRxiv. 10.1101/757252. PMC8065141 33893299 · DOI
  41. Roca X, Krainer AR, and Eperon IC (2013). Pick one, but be quick: 5′ splice sites and the problems of too many choices. Genes Dev. 27, 129– 144. 23348838 PMC3566305
  42. Rocklin GJ, Chidyausiku TM, Goreshnik I, Ford A, Houliston S, Lemak A, Carter L, Ravichandran R, Mulligan VK, Chevalier A, (2017). Global analysis of protein folding using massively parallel design, synthesis, and testing. Science 357, 168– 175. 28706065 PMC5568797
  43. Rosenberg AB, Patwardhan RP, Shendure J, and Seelig G (2015). Learning the sequence determinants of alternative splicing from millions of random sequences. Cell 163, 698– 711. 26496609
  44. Segler MHS, Kogej T, Tyrchan C, and Waller MP (2018). Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Cent. Sci 4, 120– 131. 29392184 PMC5785775
  45. Senior AW, Evans R, Jumper J, Kirkpatrick J, Sifre L, Green T, Qin C, Žídek A, Nelson AWR, Bridgland A, (2020). Improved protein structure prediction using potentials from deep learning. Nature 577, 706– 710. 31942072
  46. Shukla A, Pandey H, and Mehrotra D (2015). Comparative review of selection techniques in genetic algorithm. In International Conference on Futuristic Trends on Computational Analysis and Knowledge Management, pp. 515– 519.
  47. Stewart K, Chen Y, Ward D, Liu X, Seelig G, Strauss K, and Ceze L (2018). A content-addressable dna database with learned sequence encodings. In 24th International Conference On DNA Computing and Molecular Programming, pp. 55– 70.
  48. Tian B, and Manley JL (2017). Alternative polyadenylation of mRNA precursors. Nat. Rev. Mol. Cell Biol 18, 18– 30. 27677860 PMC5483950
  49. Wang D, Zhang C, Wang B, Li B, Wang Q, Liu D, Wang H, Zhou Y, Shi L, Lan F, and Wang Y (2019a). Optimized Crispr guide rna design for two high-fidelity cas9 variants by deep learning. Nat. Commun 10, 4284. 31537810 PMC6753114
  50. Wales David J., and Doye Jonathan P.K. (1997). Global optimization by basin-hopping and the lowest energy structures of Lennard-Jones clusters containing up to 110 atoms. The Journal of Physical Chemistry A 101 (28), 5111– 5116, 10.1021/jp970984n. · DOI
  51. Wang Y, Wang H, Liu L, and Wang X (2019b). Synthetic promoter design in Escherichia coli based on generative adversarial network. bioRxiv. 10.1101/563775. PMC7337522 32424410 · DOI
  52. Zhou J, and Troyanskaya OG (2015). Predicting effects of noncoding variants with deep learning–based sequence model. Nat. Methods 12, 931– 934. 26301843 PMC4768299

Cited by

Sorted by citations

Authors
Linder, J., Bogard, N., Rosenberg, A. B., & Seelig, G.
Affiliations
University of Washington (US)
Corresponding author
Linder, Johannes
ORCID
Venue
Cell Systems
Publisher
Elsevier BV
Published
2020-07
Volume
11
Issue
1
Pages
49-62.e16
ISSN
2405-4712 (ISSN-L)
DOI
10.1016/j.cels.2020.05.007
PMC
PMC8694568
OpenAlex
W3037592148
Cited by
107
References
12
Citations by year
2026: 8 · 2025: 18 · 2024: 19 · 2023: 16 · 2022: 22 · 2021: 19 · 2020: 4 · 2019: 1
FWCI
4.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Cell Systems (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, cc-by-nc-nd)
Funders
National Human Genome Research Institute
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › CRISPR and Genetic Engineering
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
generative neural networks, protein design, sequence diversity, fitness prediction, variational autoencoder, polyadenylation signal, differential splicing, gradient ascent optimization, local minima
MeSH terms
DNA, Humans, Neural Networks, Computer, Sequence Analysis, Protein

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review