DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome
Open accessPublished version of a preprint
The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Motivation: Deciphering the language of non-coding DNA is one of the fundamental problems in genome research. Gene regulatory code is highly complex due to the existence of polysemy and distant semantic relationship, which previous informatics methods often fail to capture especially in data-scarce scenarios. Results: To address this challenge, we developed a novel pre-trained bidirectional encoder representation, named DNABERT, to capture global and transferrable understanding of genomic DNA sequences based on up and downstream nucleotide contexts. We compared DNABERT to the most widely used programs for genome-wide regulatory elements prediction and demonstrate its ease of use, accuracy and efficiency. We show that the single pre-trained transformers model can simultaneously achieve state-of-the-art performance on prediction of promoters, splice sites and transcription factor binding sites, after easy fine-tuning using small task-specific labeled data. Further, DNABERT enables direct visualization of nucleotide-level importance and semantic relationship within input sequences for better interpretability and accurate identification of conserved sequence motifs and functional genetic variant candidates. Finally, we demonstrate that pre-trained DNABERT with human genome can even be readily applied to other organisms with exceptional performance. We anticipate that the pre-trained DNABERT model can be fined tuned to many other sequence analyses tasks. Availability and implementation: The source code, pretrained and finetuned model for DNABERT are available at GitHub (https://github.com/jerryji1993/DNABERT). Supplementary information: Supplementary data are available at Bioinformatics online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“We excluded DNABERT-1 from this comparison as it can only handle a maximum input length of 512 bp and thus could not be used for most tasks20.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [20]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, Transformer-based DNA models are constrained to short context lengths and use schemes that aggregate nucleotides into the basic units of language models, called tokens, thereby sacrificing single-nucleotide resolution (15, 16, 21–23).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In biology applications, deep learning has been employed to learn the structure of DNA and protein sequences [6,7]”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“It makes it well-suited to learn the sequence data of biology [76], including the DNA [6], RNA [77], and protein [7] sequence learning, genomic regulatory prediction [78], protein function annotation [79], protein design [80], and further adaptation into medical image [81] and gene expression analysis [13].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing (NLP)6,7 and were also recently applied to model short DNA sequences8.”
From Effective gene expression prediction from sequence by integrating long-range interactions (2021) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
An integrated encyclopedia of DNA elements in the human genome
The ENCODE Project Consortium (2012) · Nature · doi:10.1038/nature11247 · Cited by 18,072
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
dbSNP: the NCBI database of genetic variation
Sherry, S. T. (2001) · Nucleic Acids Research · doi:10.1093/nar/29.1.308 · Cited by 7,502
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Initial sequencing and comparative analysis of the mouse genome
Mouse Genome Sequencing Consortium (2002) · Nature · doi:10.1038/nature01262 · Cited by 7,187
Buenrostro, J. D., Giresi, P. G., Zaba, L. C., Chang, H. Y., & Greenleaf, W. J. (2013) · Nature Methods · doi:10.1038/nmeth.2688 · Cited by 6,422
GENCODE: The reference human genome annotation for The ENCODE Project
Harrow, J., Frankish, A., Gonzalez, J. M., Tapanari, E., Diekhans, M., Kokocinski, F., Aken, B. L., Barrell, D., Zadissa, A., Searle, S., et al. (2012) · Genome Research · doi:10.1101/gr.135350.111 · Cited by 4,585
Buniello, A., MacArthur, J. A. L., Cerezo, M., Harris, L. W., Hayhurst, J., Malangone, C., McMahon, A., Morales, J., Mountjoy, E., Sollis, E., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1120 · Cited by 4,000
ClinVar: public archive of relationships among sequence variation and human phenotype
Landrum, M. J., Lee, J. M., Riley, G. R., Jang, W., Rubinstein, W. S., Church, D. M., & Maglott, D. R. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1113 · Cited by 3,516
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873
Predicting effects of noncoding variants with deep learning–based sequence model
Zhou, J., & Troyanskaya, O. G. (2015) · Nature Methods · doi:10.1038/nmeth.3547 · Cited by 2,338
Quantifying similarity between motifs
Gupta, S., Stamatoyannopoulos, J. A., Bailey, T. L., & Noble, W. S. (2007) · Genome Biology · doi:10.1186/gb-2007-8-2-r24 · Cited by 2,058
A census of human RNA-binding proteins
Gerstberger, S., Hafner, M., & Tuschl, T. (2014) · Nature Reviews Genetics · doi:10.1038/nrg3813 · Cited by 2,032
Splicing regulation: From a parts list of regulatory elements to an integrated splicing code
Wang, Z., & Burge, C. B. (2008) · RNA · doi:10.1261/rna.876308 · Cited by 1,086
Kelley, D. R., Snoek, J., & Rinn, J. L. (2016) · Genome Research · doi:10.1101/gr.200535.115 · Cited by 1,048
Automatic annotation of eukaryotic genes, pseudogenes and promoters
Solovyev, V., Kosarev, P., Seledsov, I., & Vorobyev, D. (2006) · Genome Biology · doi:10.1186/gb-2006-7-s1-s10 · Cited by 920
A primer on deep learning in genomics
Zou, J., Huss, M., Abid, A., Mohammadi, P., Torkamani, A., & Telenti, A. (2019) · Nature Genetics · doi:10.1038/s41588-018-0295-5 · Cited by 891
Quang, D., & Xie, X. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw226 · Cited by 853
ENCODE Data in the UCSC Genome Browser: year 5 update
Rosenbloom, K. R., Sloan, C. A., Malladi, V. S., Dreszer, T. R., Learned, K., Kirkup, V. M., Wong, M. C., Maddren, M., Fang, R., Heitner, S. G., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks1172 · Cited by 766
Mapping genome-wide transcription-factor binding sites using DAP-seq
Bartlett, A., O'Malley, R. C., Huang, S.-S. C., Galli, M., Nery, J. R., Gallavotti, A., & Ecker, J. R. (2017) · Nature Protocols · doi:10.1038/nprot.2017.055 · Cited by 729
Determinants of enhancer and promoter activities of regulatory elements
Andersson, R., & Sandelin, A. (2020) · Nature Reviews Genetics · doi:10.1038/s41576-019-0173-8 · Cited by 713
Head, T. (1987) · Bulletin of Mathematical Biology · doi:10.1007/bf02481771 · Cited by 652
A Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600
An encyclopedia of mouse DNA elements (Mouse ENCODE)
Mouse ENCODE Consortium, Stamatoyannopoulos, J. A., Snyder, M., Hardison, R., Ren, B., Gingeras, T., Gilbert, D. M., Groudine, M., Bender, M., Kaul, R., et al. (2012) · Genome Biology · doi:10.1186/gb-2012-13-8-418 · Cited by 446
The functional consequences of alternative promoter use in mammalian genomes
Davuluri, R. V., Suzuki, Y., Sugano, S., Plass, C., & Huang, T. H.-M. (2008) · Trends in Genetics · doi:10.1016/j.tig.2008.01.008 · Cited by 408
Self-citationSearls, D. B. (2002) · Nature · doi:10.1038/nature01255 · Cited by 385
Cosma, M. P., Pepe, S., Annunziata, I., Newbold, R. F., Grompe, M., Parenti, G., & Ballabio, A. (2003) · Cell · doi:10.1016/s0092-8674(03)00348-9 · Cited by 365
RNA codewords and protein synthesis, VII. On the general nature of the RNA code.
Nirenberg, M., Leder, P., Bernfield, M., Brimacombe, R., Trupin, J., Rottman, F., & O'Neal, C. (1965) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.53.5.1161 · Cited by 334
Linguistic Features of Noncoding DNA Sequences
Mantegna, R. N., Buldyrev, S. V., Goldberger, A. L., Havlin, S., Peng, C. K., Simons, M., & Stanley, H. E. (1994) · Physical Review Letters · doi:10.1103/physrevlett.73.3169 · Cited by 306
The Landscape of Isoform Switches in Human Cancers
Vitting-Seerup, K., & Sandelin, A. (2017) · Molecular Cancer Research · doi:10.1158/1541-7786.mcr-16-0459 · Cited by 256
Leslie, R., O’Donnell, C. J., & Johnson, A. D. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu273 · Cited by 254
Recurrent Neural Network for Predicting Transcription Factor Binding Sites
Shen, Z., Bao, W., & Huang, D.-S. (2018) · Scientific Reports · doi:10.1038/s41598-018-33321-1 · Cited by 211
Head, T. (1987) · Bulletin of Mathematical Biology · doi:10.1016/s0092-8240(87)90018-8 · Cited by 193
DeePromoter: Robust Promoter Predictor Using Deep Learning
Oubounyt, M., Louadi, Z., Tayara, H., & Chong, K. T. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.00286 · Cited by 191
EPD and EPDnew, high-quality promoter resources in the next-generation sequencing era
Dreos, R., Ambrosini, G., Cavin Périer, R., & Bucher, P. (2013) · Nucleic Acids Research · doi:10.1093/nar/gks1233 · Cited by 168
DeeperBind: Enhancing prediction of sequence specificities of DNA binding proteins
Hassanzadeh, H. R., & Wang, M. D. (2016) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822515 · Cited by 167
Li, F., Jin, Y., Liu, W., Rawat, B. P. S., Cai, P., & Yu, H. (2019) · JMIR Medical Informatics · doi:10.2196/14830 · Cited by 161
DeepSite: bidirectional LSTM and CNN models for predicting DNA–protein binding
Zhang, Y., Qiao, S., Ji, S., & Li, Y. (2019) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-019-00990-x · Cited by 134
Searls, D. B. (1992) · American Scientist · Cited by 126
Promoter analysis and prediction in the human genome using sequence-based deep learning models
Umarov, R. K., Kuwahara, H., Li, Y., Gao, X., & Solovyev, V. V. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty1068 · Cited by 120
A Simple Hyper-Geometric Approach for Discovering Putative Transcription Factor Binding Sites
Barash, Y., Bejerano, G., & Friedman, N. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44696-6_22 · Cited by 99
Yoon, H., Liyanarachchi, S., Wright, F. A., Davuluri, R. V., Lockman, J. C., de la Chapelle, A., & Pellegata, N. S. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.242597299 · Cited by 95
Self-citationGenome structure described by formal languages
Brendel, V., & Busse, H. G. (1984) · Nucleic Acids Research · doi:10.1093/nar/12.5.2561 · Cited by 85
SpliceFinder: ab initio prediction of splice sites using convolutional neural network
Wang, R., Wang, Z., Wang, J., & Li, S. C. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3306-3 · Cited by 83
The Linguistics of DNA: Words, Sentences, Grammar, Phonetics, and Semantics
Ji, S. (1999) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1999.tb08916.x · Cited by 70
MYO7A mutation screening in Usher syndrome type I patients from diverse origins
Jaijo, T., Aller, E., Beneyto, M., Najera, C., Graziano, C., Turchetti, D., Seri, M., Ayuso, C., Baiget, M., Moreno, F., et al. (2007) · Journal of Medical Genetics · doi:10.1136/jmg.2006.045377 · Cited by 70
A novel method for improved accuracy of transcription factor binding site prediction
Khamis, A. M., Motwalli, O., Oliva, R., Janković, B. R., Medvedeva, Y. A., Ashoor, H., Essack, M., Gao, X., & Bajić, V. B. (2018) · Nucleic Acids Research · doi:10.1093/nar/gky237 · Cited by 62
Crosstalk between c-Jun and TAp73α/β contributes to the apoptosis–survival balance
Koeppel, M., van Heeringen, S. J., Kramer, D., Smeenk, L., Janssen‐Megens, E. M., Hartmann, M., Stunnenberg, H. G., & Lohrum, M. A. E. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr028 · Cited by 60
Novel Transformer Networks for Improved Sequence Labeling in genomics
Clauwaert, J., & Waegeman, W. (2020) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2020.3035021 · Cited by 50
Predicting regulatory variants with composite statistic
Li, M. J., Pan, Z., Liu, Z., Wu, J., Wang, P., Zhu, Y., Xu, F., Xia, Z., Sham, P. C., Kocher, J.-P. A., et al. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw288 · Cited by 41
The context of gene expression regulation
Gibcus, J. H., & Dekker, J. (2012) · F1000 Biology Reports · doi:10.3410/b4-8 · Cited by 41
In silico analysis of alternative splicing on drug-target gene interactions
Ji, Y., Mishra, R. K., & Davuluri, R. V. (2020) · Scientific Reports · doi:10.1038/s41598-019-56894-x · Cited by 27
Self-citationLiang, X., Yan, D., Zhao, J., Ding, W., Xu, X., & Wang, X. (2018) · Oncology Letters · doi:10.3892/ol.2018.9350 · Cited by 8
Application of First EF to Find Promoters and First Exons in the Human Genome
Davuluri, R. V. (2003) · Current Protocols in Bioinformatics · doi:10.1002/0471250953.bi0407s01 · Cited by 7
Self-citation
Research works printed but not linked: 1
- Yang, 2019, pp. 5754
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Effective gene expression prediction from sequence by integrating long-range interactions
Avsec, Ž., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., & Kelley, D. R. (2021) · Nature Methods · doi:10.1038/s41592-021-01252-x · Cited by 1,326
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 772
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Chai-1: Decoding the molecular interactions of life
Discovery, C., Boitreaud, J., Dent, J., McPartlon, M., Meier, J., Reis, V., Rogozhnikov, A., & Wu, K. (2024) · doi:10.1101/2024.10.10.615955 · Cited by 517
Metadata
- Authors
- Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V.
- Affiliations
- Northwestern University (US)
- Stony Brook University (US)
- Corresponding author
- Liu, Han
- Davuluri, Ramana V
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-02-04
- Volume
- 37
- Issue
- 15
- Pages
- 2112-2120
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC11025658
- OpenAlex
- W3127238141
- Cited by
- 1,303
- References
- 58
- Citations by year
- 2026: 379 · 2025: 406 · 2024: 269 · 2023: 146 · 2022: 78 · 2021: 25
- FWCI
- 41.61
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, other-oa)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Library of Medicine
- National Institutes of Health (R01LM011297)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- DNABERT, DNA sequence representation, pre-trained transformer models, non-coding DNA, promoter identification, splice site prediction, transcription factor binding sites, regulatory element prediction, motif discovery, functional variants, interspecies transmission, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.