GENA-LM: a family of open-source foundational DNA language models for long sequences
Open accessPublished version of a preprint
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Recent advancements in genomics, propelled by artificial intelligence, have unlocked unprecedented capabilities in interpreting genomic sequences, mitigating the need for exhaustive experimental analysis of complex, intertwined molecular processes inherent in DNA function. A significant challenge, however, resides in accurately decoding genomic sequences, which inherently involves comprehending rich contextual information dispersed across thousands of nucleotides. To address this need, we introduce GENA language model (GENA-LM), a suite of transformer-based foundational DNA language models capable of handling input lengths up to 36 000 base pairs. Notably, integrating the newly developed recurrent memory mechanism allows these models to process even larger DNA segments. We provide pre-trained versions of GENA-LM, including multispecies and taxon-specific models, demonstrating their capability for fine-tuning and addressing a spectrum of complex biological tasks with modest computational demands. While language models have already achieved significant breakthroughs in protein biology, GENA-LM showcases a similarly promising potential for reshaping the landscape of genomics and multi-omics data analysis. All models are publicly available on GitHub (https://github.com/AIRI-Institute/GENA_LM) and on HuggingFace (https://huggingface.co/AIRI-Institute). In addition, we provide a web service (https://dnalm.airi.net/) allowing user-friendly DNA annotation with GENA-LM models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435
Effective gene expression prediction from sequence by integrating long-range interactions
Avsec, Ž., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., & Kelley, D. R. (2021) · Nature Methods · doi:10.1038/s41592-021-01252-x · Cited by 1,326
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 772
The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., De Almeida, B. P., Sirelkhatim, H., et al. (2023) · doi:10.1101/2023.01.11.523679 · Cited by 680
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
DeepSTARR predicts enhancer activity from DNA sequence and enables the de novo design of enhancers
de Almeida, B. P., Reiter, F., Pagani, M., & Stark, A. (2021) · doi:10.1101/2021.10.05.463203 · Cited by 284
de Almeida, B. P., Reiter, F., Pagani, M., & Stark, A. (2022) · Nature Genetics · doi:10.1038/s41588-022-01048-5 · Cited by 283
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., & Liu, H. (2023) · arXiv · doi:10.48550/arxiv.2306.15006 · Cited by 179
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Nguyen, E., Poli, M., Faizi, M., Thomas, A., Birch-Sykes, C., Wornow, M., Patel, A., Rabideau, C., Massaroli, S., Bengio, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.15794 · Cited by 168
Memformer: a memory-augmented transformer for sequence modeling
Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., & Yu, Z. (2020) · arXiv · doi:10.48550/arxiv.2010.06891 · Cited by 80
Explainability in transformer models for functional genomics
Clauwaert, J., Menschaert, G., & Waegeman, W. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab060 · Cited by 60
Bulatov, A., Kuratov, Y., & Burtsev, M. S. (2022) · arXiv · doi:10.48550/arxiv.2207.06881 · Cited by 55
Self-citationHutchins, D., Schlag, I., Wu, Y., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2203.07852 · Cited by 43
Scaling Transformer to 1M tokens and beyond with RMT
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.11062 · Cited by 19
Self-citationBeyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12
Self-citation
Research works printed but not linked: 38
- Kim, 2023, Deciphering the multi-scale, quantitative cis-regulatory code, Mol. Cell, 83, 373, 10.1016/j.molcel.2022.12.032 · DOI · Paper page
- Whalen, 2022, Navigating the pitfalls of applying machine learning in genomics, Nat. Rev. Genet., 23, 169, 10.1038/s41576-021-00434-9 · DOI · Paper page
- Libbrecht, 2015, Machine learning applications in genetics and genomics, Nat. Rev. Genet., 16, 321, 10.1038/nrg3920 · DOI · Paper page
- Belokopytova, 2020, Quantitative prediction of enhancer–promoter interactions, Genome Res., 30, 72, 10.1101/gr.249367.119 · DOI · Paper page
- Sindeeva M., Chekanov N., Avetisian M., Shashkova T.I., Baranov N., Malkin E., Lapin A., Kardymon O., Fishman V. Cell type-specific interpretation of noncoding variants using deep learning-based methods. GigaScience. 2023; 12:giad015. · DOI
- Belokopytova, 2021, Predicting genome architecture: challenges and solutions, Front. Genet., 11, 617202, 10.3389/fgene.2020.617202 · DOI
- Pan, 2010, A survey on transfer learning, IEEE Trans. Knowl. Data Eng., 22, 1345, 10.1109/TKDE.2009.191 · DOI · Paper page
- Dai, 2015, Semi-supervised sequence learning, 3079
- Ji, 2021, DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome, Bioinformatics, 37, 2112, 10.1093/bioinformatics/btab083 · DOI · Paper page
- Vaswani, 2017, Attention is all you need, 5998
- Guo, 2019, Star-transformer, 10.18653/v1/N19-1133 · DOI · Paper page
- Ainslie, 2020, ETC: encoding long and structured data in transformers, 268, 10.18653/v1/2020.emnlp-main.19 · DOI · Paper page
- Zaheer, 2020, Big Bird: transformers for longer sequences, 17283
- Choromanski, 2021, Rethinking attention with performers
- Kang, 2023, The pan-genome and local adaptation of Arabidopsis thaliana, Nat. Commun., 14, 6259, 10.1038/s41467-023-42029-4 · DOI
- O’Donnell, 2023, Telomere-to-telomere assemblies of 142 strains characterize the genome structural landscape in Saccharomyces cerevisiae, Nat. Genet., 55, 1390, 10.1038/s41588-023-01459-y · DOI · Paper page
- Kim, 2024, Single-fly assemblies fill major phylogenomic gaps across the Drosophilidae Tree of Life, PLoS Biol., 22, e3002697, 10.1371/journal.pbio.3002697 · DOI
- Jaganathan, 2019, Predicting splicing from primary sequence with deep learning, Cell, 176, 535, 10.1016/j.cell.2018.12.015 · DOI · Paper page
- Zhou, 2015, Predicting effects of noncoding variants with deep learning-based sequence model, Nat. Methods, 12, 931, 10.1038/nmeth.3547 · DOI · Paper page
- Bogard, 2019, A deep neural network for predicting and engineering alternative polyadenylation, Cell, 178, 91, 10.1016/j.cell.2019.04.046 · DOI · Paper page
- Xiong, 2020, On layer normalization in the transformer architecture, 10524
- Loshchilov, 2019, Decoupled weight decay regularization
- Mukund, 2017, Axiomatic attribution for deep networks, 3319
- Grant, 2021, XSTREME: comprehensive motif analysis of biological sequence datasets, 10.1101/2021.09.02.458722 · DOI
- Kulakovskiy, 2018, HOCOMOCO: towards a complete collection of transcription factor binding models for human and mouse via large-scale ChIP-Seq analysis, Nucleic Acids Res., 46, D252, 10.1093/nar/gkx1106 · DOI · Paper page
- Rhee, 2012, ChIP-exo: a method to identify genomic location of DNA-binding proteins at near single nucleotide accuracy
- Kawana, 1995, Cooperative interaction of GATA-2 and AP1 regulates transcription of the endothelin-1 gene, Mol. Cell. Biol., 15, 4225, 10.1128/MCB.15.8.4225 · DOI · Paper page
- Menendez-Gonzalez, 2019, Inhibition of GATA2 restrains cell proliferation and enhances apoptosis and chemotherapy mediated apoptosis in human GATA2 overexpressing AML cells, Sci. Rep., 9, 12212, 10.1038/s41598-019-48589-0 · DOI
- Eferl, 2003, AP-1: a double-edged sword in tumorigenesis, Nat. Rev. Cancer, 3, 859, 10.1038/nrc1209 · DOI · Paper page
- Camellato, 2024, Synthetic reversed sequences reveal default genomic states, Nature, 628, 373, 10.1038/s41586-024-07128-2 · DOI
- Shmelev, 2024, GENA-Web - GENomic Annotations Web Inference using DNA language models, 10.1101/2024.04.26.591391 · DOI
- Zhang, 2022, Transformer for gene expression modeling (T-GEM): an interpretable deep learning model for gene expression-based phenotype predictions, Cancers, 14, 4763, 10.3390/cancers14194763 · DOI
- Le, 2022, Deep transformers and convolutional neural network in identifying DNA N6-methyladenine sites in cross-species genomes, Methods, 204, 199, 10.1016/j.ymeth.2021.12.004 · DOI
- Kabirova, 2023, Function and evolution of the loop extrusion machinery in animals, Int. J. Mol. Sci., 24, 5017, 10.3390/ijms24055017 · DOI
- Fishman, 2018, Interpreting chromosomal rearrangements in the context of 3-dimentional genome organization: a practical guide for medical genetics, Biochemistry (Mosc.), 83, 393, 10.1134/S0006297918040107 · DOI
- Shashkova, 2022, SEMA: antigen B-cell conformational epitope prediction using deep transfer learning, Front. Immunol., 13, 960985, 10.3389/fimmu.2022.960985 · DOI · Paper page
- Madani, 2023, Large language models generate functional protein sequences across diverse families, Nat. Biotechnol., 41, 1099, 10.1038/s41587-022-01618-2 · DOI · Paper page
- Wang, 2022, LM-GVP: an extensible sequence and structure informed deep learning framework for protein property prediction, Sci. Rep., 12, 6832, 10.1038/s41598-022-10775-y · DOI · Paper page
Cited by
The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., De Almeida, B. P., Sirelkhatim, H., et al. (2023) · doi:10.1101/2023.01.11.523679 · Cited by 680
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 772
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Xia, Y., Du, X., Liu, B., Guo, S., & Huo, Y.-X. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae429 · Cited by 31
GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction
Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2023) · doi:10.1101/2023.10.10.561776 · Cited by 118
Metadata
- Authors
- Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M.
- Venue
- Nucleic Acids Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-01-11
- Volume
- 53
- Issue
- 2
- ISSN
- 0305-1048 (ISSN-L)
- 1362-4962
- OpenAlex
- W4406419344
- Cited by
- 135
- References
- 27
- FWCI
- 15.34
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- AIRI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.