CoChat
Sign Up
PreprintCited by 12Machine Learning in BioinformaticsGenomics and Chromatin DynamicsFractal and DNA sequence analysis

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Self-supervised learning for DNA sequences with circular dilated convolutional networks

Open access
Cheng, Lei, Yu, Tong, Aittokallio, Tero, Corander, Jukka, Khalitov, Ruslan, Yang, Zhirong·2023·Neural Networks
DOI 10.1101/2023.01.30.526193

Preprint — published as 10.1016/j.neunet.2023.12.002 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Due to their intrinsic properties, DNA molecules commonly exhibit long-range interactions along a linear sequence representation. Taking this information into account when modeling DNA sequences is therefore important for obtaining more accurate sequence-based inference. Many deep learning methods have recently been developed for this purpose, but they still suffer from two major issues. First, the existing methods can only handle short DNA fragments, thereby losing longerrange interactions. Second, the current methods require massive supervised labeling while missing most order information within the sequences. Consequently, there is a need to develop an efficient deep neural network modeling framework to extract wide contextual information for more accurate sequence-based inference tasks. Our new framework, named Revolution, takes full DNA sequences as input, without any condensation, and can give accurate predictions for DNA sequences up to 10kbp. In variant effect prediction, our method increases the Area Under the Receiver Operating Characteristics (AUROC) by 19.61% on 49 human tissues on average. Revolution is also demonstrated to work on the plant sequences by improving 2.36% AUROC on average for predicting open chromatin regions (OCRs). The data, models, and code can be freely accessed at https://github.com/wiedersehne/Revolution-DNAPretraining .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.01.30.526193

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Temporal Convolutional Networks for Action Segmentation and Detection

    Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · cited 1×

    “The dilated convolutions can achieve large receptive fields in a few layers (Oord et al., 2016; Kalchbrenner et al., 2016; Lea et al., 2017; Bai et al., 2018).”
    Background

  • Neural Machine Translation in Linear Time

    Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · cited 1×

    “The dilated convolutions can achieve large receptive fields in a few layers (Oord et al., 2016; Kalchbrenner et al., 2016; Lea et al., 2017; Bai et al., 2018).”
    Background

Flagged first, then by citations

Research works printed but not linked: 39

  1. 2019, CNN-MGP: convolutional neural networks for metagenomics gene prediction, Interdisciplinary Sciences: Computational Life Sciences, 11, 628
  2. 10.1038/nbt.3300 · DOI · Paper page
  3. Alsallakh, B. , Kokhlikyan, N. , Miglani, V. , Yuan, J. , and Reblitz-Richardson, O. Mind the pad–CNNs can develop blind spots. arXiv preprint arXiv:2010.02178, 2020. · arXiv
  4. An, W. , Guo, Y. , Bian, Y. , Ma, H. , Yang, J. , Li, C. , and Huang, J. MoDNA: motif-oriented pre-training for DNA language model. In Proceedings of the 13th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pages 1–5, 2022. · DOI
  5. 2021, Effective gene expression prediction from sequence by integrating long-range interactions, Nature methods, 18, 1196, 10.1038/s41592-021-01252-x · DOI · Paper page
  6. Bahdanau, D. , Cho, K. , and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. · arXiv
  7. Bai, S. , Kolter, J. Z. , and Koltun, V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018. · arXiv
  8. 2013, Transposition of native chromatin for fast and sensitive epigenomic profiling of open chromatin, DNA-binding proteins and nucleosome position, Nature methods, 10, 1213, 10.1038/nmeth.2688 · DOI · Paper page
  9. 10.1038/s41592-019-0360-8 · DOI
  10. 2023, Classification of long sequential data using circular dilated convolutional neural networks, Neurocomputing, 518, 50, 10.1016/j.neucom.2022.10.054 · DOI
  11. 2012, Enhancers, Wiley Interdisciplinary Reviews: Developmental Biology, 1, 469, 10.1002/wdev.53 · DOI
  12. Devlin, J. , Chang, M.-W. , Lee, K. , and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
  13. Gunasekaran, H. , Ramalakshmi, K. , Rex Macedo Arokiaraj, A. , Deepa Kanmani, S. , Venkatesan, C. , and Suresh Gnana Dhas, C. Analysis of DNA sequence classification using CNN and hybrid models. Computational and Mathematical Methods in Medicine, 2021, 2021. · DOI
  14. He, K. , Chen, X. , Xie, S. , Li, Y. , Dollár, P. , and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022. · DOI · Paper page
  15. 10.1162/neco.1997.9.8.1735 · DOI · Paper page
  16. 2021, DNABERT: pre-trained bidirectional encoder representations from Transformers model for DNA-language in genome, Bioinformatics, 37, 2112, 10.1093/bioinformatics/btab083 · DOI · Paper page
  17. Kayhan, O. S. and Gemert, J. C. v. On translation invariance in CNNs: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14274–14285, 2020.
  18. 10.1101/gr.227819.117 · DOI · Paper page
  19. 2020, Cross-species regulatory sequence activity prediction, PLoS computational biology, 16, e1008050, 10.1371/journal.pcbi.1008050 · DOI · Paper page
  20. Khalitov, R. , Yu, T. , Cheng, L. , and Yang, Z. ChordMixer: A scalable neural attention model for sequences with different lengths. arXiv preprint arXiv:2206.05852, 2022. · arXiv
  21. Kumar, A. , Raghunathan, A. , Jones, R. , Ma, T. , and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022. · arXiv
  22. 10.1038/ng.3331 · DOI
  23. 2019, Detection of DNA base modifications by deep recurrent neural network on Oxford Nanopore sequencing data, Nature communications, 10, 1
  24. 2021, Predicting enhancer-promoter interactions by deep learning and matching heuristic, Briefings in Bioinformatics, 22, bbaa254, 10.1093/bib/bbaa254 · DOI
  25. Mo, S. , Fu, X. , Hong, C. , Chen, Y. , Zheng, Y. , Tang, X. , Shen, Z. , Xing, E. P. , and Lan, Y. Multi-modal self-supervised pre-training for regulatory genome across cell types. arXiv preprint arXiv:2110.05231, 2021. · arXiv
  26. Oord, A. v. d. , Dieleman, S. , Zen, H. , Simonyan, K. , Vinyals, O. , Graves, A. , Kalchbrenner, N. , Senior, A. , and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016. · arXiv
  27. Rizzo, R. , Fiannaca, A. , La Rosa, M. , and Urso, A. A deep learning approach to DNA sequence classification. In International Meeting on Computational Intelligence Methods for Bioinformatics and Biostatistics, pages 129–140. Springer, 2015. · DOI
  28. 2020, Supervised enhancer prediction with epigenetic pattern recognition and targeted validation, Nature methods, 17, 807, 10.1038/s41592-020-0907-8 · DOI · Paper page
  29. 1953, “Cloze procedure”: A new tool for measuring readability, Journalism quarterly, 30, 415, 10.1177/107769905303000401 · DOI · Paper page
  30. 10.1007/978-3-030-91415-8_18 · DOI
  31. 2020, Functionally informed fine-mapping and polygenic localization of complex trait heritability, Nature genetics, 52, 1355, 10.1038/s41588-020-00735-5 · DOI · Paper page
  32. Wettig, A. , Gao, T. , Zhong, Z. , and Chen, D. Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.08005, 2022. · arXiv
  33. 10.1093/bioinformatics/btx105 · DOI
  34. Yu, T. , Khalitov, R. , Cheng, L. , and Yang, Z. Paramixer: Parameterizing mixing links in sparse factors works better than dot-product self-attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 691–700, 2022. · DOI
  35. 2020, DeepSite: bidirectional LSTM and CNN models for predicting DNA–protein binding, International Journal of Machine Learning and Cybernetics, 11, 841, 10.1007/s13042-019-00990-x · DOI
  36. 2021, PlantDeepSEA, a deep learning-based web service to predict the regulatory effects of genomic variants in plants, Nucleic Acids Research, 49, W523, 10.1093/nar/gkab383 · DOI
  37. 2015, Predicting effects of noncoding variants with deep learning– based sequence model, Nature methods, 12, 931, 10.1038/nmeth.3547 · DOI · Paper page
  38. 10.1093/bioinformatics/bty1050 · DOI
  39. 2021, PlantDeepSEA, a deep learning-based web service to predict the regulatory effects of genomic variants in plants, Nucleic Acids Research, 49, W523, 10.1093/nar/gkab383 · DOI

Metadata

Authors
Cheng, L., Yu, T., Aittokallio, T., Corander, J., Khalitov, R., & Yang, Z.
Affiliations
  • Norwegian University of Science and Technology (NO)
  • Oslo University Hospital (NO)
  • University of Oslo (NO)
Corresponding author
Yang, Zhirong
ORCID
Venue
Neural Networks
Publisher
openRxiv
Published
2023-02-01
Volume
171
Pages
466-473
ISSN
0893-6080 (ISSN-L)
DOI
10.1101/2023.01.30.526193
OpenAlex
W4389262896
Cited by
12
References
2
Citations by year
2026: 2 · 2025: 4 · 2024: 5 · 2023: 1
FWCI
1.07
Type
journal article
Language
en
Open access
Locations
  • Neural Networks (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed (publishedVersion)
Funders
  • Norges Forskningsråd (287284)
  • Finnish Center for Artificial Intelligence
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
Keywords
DNA sequence modeling, self-supervised learning, long-range interactions, nucleobase interactions, plant chromatin
MeSH terms
Supervised Machine Learning, Base Sequence, DNA, Humans, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review