Self-supervised Learning for DNA sequences with circular dilated convolutional networks
Open accessPublished version of a preprint
Abstract
DNA molecules commonly exhibit wide interactions between the nucleobases. Modeling the interactions is important for obtaining accurate sequence-based inference. Although many deep learning methods have recently been developed for modeling DNA sequences, they still suffer from two major issues: 1) most existing methods can handle only short DNA fragments and fail to capture long-range information; 2) current methods always require massive supervised labels, which are hard to obtain in practice. We propose a new method to address both issues. Our neural network employs circular dilated convolutions as building blocks in the backbone. As a result, our network can take long DNA sequences as input without any condensation. We also incorporate the neural network into a self-supervised learning framework to capture inherent information in DNA without expensive supervised labeling. We have tested our model in two DNA inference tasks, the human variant effect and the open chromatin region of plants, where the experimental results show that our method outperforms five other deep learning models. Our code is available at https://github.com/wiedersehne/cdilDNA.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · see citing works
Buenrostro, J. D., Giresi, P. G., Zaba, L. C., Chang, H. Y., & Greenleaf, W. J. (2013) · Nature Methods · doi:10.1038/nmeth.2688 · Cited by 6,422 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1803.01271 · Cited by 4,655 · see citing works
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873 · see citing works
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · see citing works
Predicting effects of noncoding variants with deep learning–based sequence model
Zhou, J., & Troyanskaya, O. G. (2015) · Nature Methods · doi:10.1038/nmeth.3547 · Cited by 2,338 · see citing works
“Cloze Procedure”: A New Tool for Measuring Readability
Taylor, W. L. (1953) · Journalism Quarterly · doi:10.1177/107769905303000401 · Cited by 2,332 · see citing works
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238 · see citing works
Effective gene expression prediction from sequence by integrating long-range interactions
Avsec, Ž., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., & Kelley, D. R. (2021) · Nature Methods · doi:10.1038/s41592-021-01252-x · Cited by 1,326 · see citing works
Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab083 · Cited by 1,303 · see citing works
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionPreprint
Kumar, A., Raghunathan, A., Jones, R., Ma, T.-Y., & Liang, P. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2202.10054 · Cited by 1,110 · see citing works
Sequential regulatory activity prediction across chromosomes with convolutional neural networks
Kelley, D. R., Reshef, Y. A., Bileschi, M., Belanger, D., McLean, C. Y., & Snoek, J. (2018) · Genome Research · doi:10.1101/gr.227819.117 · Cited by 537 · see citing works
A method to predict the impact of regulatory variants from DNA sequence
Lee, D., Gorkin, D. U., Baker, M., Strober, B. J., Asoni, A. L., McCallion, A. S., & Beer, M. (2015) · Nature Genetics · doi:10.1038/ng.3331 · Cited by 417 · see citing works
Neural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · see citing works
Functionally informed fine-mapping and polygenic localization of complex trait heritability
Weissbrod, O., Hormozdiari, F., Benner, C., Cui, R., Ulirsch, J., Gazal, S., Schoech, A. P., van de Geijn, B., Reshef, Y., Márquez-Luna, C., et al. (2020) · Nature Genetics · doi:10.1038/s41588-020-00735-5 · Cited by 364 · see citing works
Liu, Q., Li, F., Yu, G., Wang, D., Xiao, C., & Wang, K. (2019) · Nature Communications · doi:10.1038/s41467-019-10168-2 · Cited by 326 · see citing works
Cross-species regulatory sequence activity prediction
Kelley, D. R. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1008050 · Cited by 228 · see citing works
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location
Kayhan, O. S., & van Gemert, J. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01428 · Cited by 215 · see citing works
Should You Mask 15% in Masked Language Modeling?
Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.217 · Cited by 161 · see citing works
Analysis of DNA Sequence Classification Using CNN and Hybrid Models
Gunasekaran, H., Ramalakshmi, K., Arokiaraj, A. R. M., Kanmani, S. D., Venkatesan, C., & Dhas, C. S. G. (2021) · Computational and Mathematical Methods in Medicine · doi:10.1155/2021/1835056 · Cited by 153 · see citing works
BiRen: predicting enhancers with a deep-learning-based model using the DNA sequence alone
Yang, B., Liu, F., Ren, C., Ouyang, Z., Xie, Z., Bo, X., & Shu, W. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx105 · Cited by 136 · see citing works
Selene: a PyTorch-based deep learning library for sequence data
Chen, K., Cofer, E. M., Zhou, J., & Troyanskaya, O. G. (2019) · Nature Methods · doi:10.1038/s41592-019-0360-8 · Cited by 132 · see citing works
DeepSite: bidirectional LSTM and CNN models for predicting DNA–protein binding
Zhang, Y., Qiao, S., Ji, S., & Li, Y. (2019) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-019-00990-x · Cited by 122 · see citing works
Supervised enhancer prediction with epigenetic pattern recognition and targeted validation
Sethi, A., Gu, M., Gumusgoz, E., Chan, L., Yan, K.-K., Rozowsky, J., Barozzi, I., Afzal, V., Akiyama, J. A., Plajzer-Frick, I., et al. (2020) · Nature Methods · doi:10.1038/s41592-020-0907-8 · Cited by 109 · see citing works
EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research InfrastructurePreprint
Själander, M., Jahre, M., Tufte, G., & Reißmann, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05848 · Cited by 94 · see citing works
Zhuang, Z., Shen, X., & Pan, W. (2019) · Bioinformatics · doi:10.1093/bioinformatics/bty1050 · Cited by 89 · see citing works
CNN-MGP: Convolutional Neural Networks for Metagenomics Gene Prediction
Al-Ajlan, A., & Allali, A. E. (2018) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-018-0313-4 · Cited by 79 · see citing works
A Deep Learning Approach to DNA Sequence Classification
Rizzo, R., Fiannaca, A., Rosa, M. L., & Urso, A. M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44332-4_10 · Cited by 76 · see citing works
Predicting enhancer-promoter interactions by deep learning and matching heuristic
Min, X., Ye, C., Liu, X., & Zeng, X. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa254 · Cited by 64 · see citing works
Zhao, H., Tu, Z., Liu, Y., Zong, Z., Li, J., Liu, H., Xiong, F., Zhan, J., Hu, X., & Xie, W. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab383 · Cited by 64 · see citing works
Mind the Pad -- CNNs can Develop Blind SpotsPreprint
Alsallakh, B., Kokhlikyan, N., Miglani, V., Yuan, J., & Reblitz-Richardson, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02178 · Cited by 55 · see citing works
Classification of long sequential data using circular dilated convolutional neural networks
Cheng, L., Khalitov, R., Yu, T., Zhang, J., & Yang, Z. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.054 · Cited by 47 · see citing works
Self-citationAn, W., Guo, Y., Bian, Y., Ma, H., Yang, J., Li, C., & Huang, J. (2022) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3535508.3545512 · Cited by 27 · see citing works
Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell TypesPreprint
Mo, S., Fu, X., Hong, C., Chen, Y., Zheng, Y., Tang, X., Shen, Z., Xing, E. P., & Lan, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05231 · Cited by 25 · see citing works
ChordMixer: A Scalable Neural Attention Model for Sequences with Different LengthsPreprint
Khalitov, R., Yu, T., Cheng, L., & Yang, Z. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2206.05852 · Cited by 16 · see citing works
Self-citationYu, T., Khalitov, R., Cheng, L., & Yang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00077 · Cited by 11 · see citing works
Self-citationCho, K. W. Y. (2012) · Wiley Interdisciplinary Reviews Developmental Biology · doi:10.1002/wdev.53 · Cited by 10 · see citing works
Wang, Z., Tan, X., Li, B., Liu, Y., Shao, Q., Li, Z., Yang, Y., & Zhang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91415-8_18 · Cited by 5 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
3 of the 45 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Zhang, L., Liu, J., Wei, Y., An, D. U., & Ning, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102814 · Cited by 90 · see citing works
BEND: Benchmarking DNA Language Models on biologically meaningful tasksPreprint
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.12570 · Cited by 101 · see citing works
Guo, Y., Xin, Y., Cao, J., Xu, Y., & Zhou, D. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111222 · Cited by 16 · see citing works
MIFS: An adaptive multipath information fused self-supervised framework for drug discovery
Gong, X., Liu, Q., Han, R., Guo, Y., & Wang, G. (2025) · Neural Networks · doi:10.1016/j.neunet.2024.107088 · Cited by 7 · see citing works
Self-distillation improves self-supervised learning for DNA sequence inference
Yu, T., Cheng, L., Khalitov, R., Olsson, E. B., & Yang, Z. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106978 · Cited by 6 · see citing works
Self-citationAdaptive batch-fusion self-supervised learning for ultrasound image pretraining
Zhang, J., Wu, X., Liu, S., Fan, Y., Chen, Y., Lyu, G., Liu, P., Liu, Z., & He, S. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102599 · Cited by 5 · see citing works
Language Modelling Techniques for Analysing the Impact of Human Genetic Variation
Hegde, M., Nebel, J., & Rahman, F. (2025) · Bioinformatics and Biology Insights · doi:10.1177/11779322251358314 · Cited by 5 · see citing works
ASWE: Adaptive Small-World Encoder for Efficient Channel Coding
Dwivedi, S., Pandey, O. J., & Hegde, R. M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464671 · Cited by 1 · see citing works
A sparse and wide neural network model for DNA sequences
Yu, T., Cheng, L., Khalitov, R., & Yang, Z. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107040 · Cited by 1 · see citing works
Self-citationSelf-Distillation Improves DNA Sequence InferencePreprint
Yu, T., Cheng, L., Khalitov, R., Olsson, E. B., & Yang, Z. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862343
Self-citationAn Application of Residue Number System (RNS) to a Next-Generation Sequencing -SOLiDPreprint
Akanbasiam, J. A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4870059
Artificial intelligence integration in livestock epigenomics
Hossein‐Zadeh, N. G. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45663-3.00005-1
- Authors
- Cheng, L., Yu, T., Khalitov, R., & Yang, Z.
- Affiliations
- Norwegian University of Science and Technology (NO)
- Corresponding author
- Yang, Zhirong
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2024-03
- Volume
- 171
- Pages
- 466-473
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W4389262896
- Cited by
- 12
- References
- 45
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 5 · 2023: 1
- FWCI
- 1.07
- Type
- journal article
- Language
- en
- Open access
- green
- dx.doi.org
- Licence
- Locations
- Neural Networks (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed (publishedVersion)
- Funders
- Norges Forskningsråd (287284)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
- Keywords
- DNA sequence modeling, self-supervised learning, long-range interactions, nucleobase interactions, plant chromatin
- MeSH terms
- Supervised Machine Learning, Base Sequence, DNA, Humans, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
