CoChat

GENA-LM: a family of open-source foundational DNA language models for long sequences

Open access
Fishman, Veniamin, Kuratov, Yuri, Shmelev, Aleksei, Petrov, Maxim, Penzar, Dmitry, Shepelin, Denis, Chekanov, Nikolay, Kardymon, Olga, Burtsev, Mikhail · 2025 · Nucleic Acids Research

Published version of a preprint

Cited by 135

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

Abstract

Recent advancements in genomics, propelled by artificial intelligence, have unlocked unprecedented capabilities in interpreting genomic sequences, mitigating the need for exhaustive experimental analysis of complex, intertwined molecular processes inherent in DNA function. A significant challenge, however, resides in accurately decoding genomic sequences, which inherently involves comprehending rich contextual information dispersed across thousands of nucleotides. To address this need, we introduce GENA language model (GENA-LM), a suite of transformer-based foundational DNA language models capable of handling input lengths up to 36 000 base pairs. Notably, integrating the newly developed recurrent memory mechanism allows these models to process even larger DNA segments. We provide pre-trained versions of GENA-LM, including multispecies and taxon-specific models, demonstrating their capability for fine-tuning and addressing a spectrum of complex biological tasks with modest computational demands. While language models have already achieved significant breakthroughs in protein biology, GENA-LM showcases a similarly promising potential for reshaping the landscape of genomics and multi-omics data analysis. All models are publicly available on GitHub (https://github.com/AIRI-Institute/GENA_LM) and on HuggingFace (https://huggingface.co/AIRI-Institute). In addition, we provide a web service (https://dnalm.airi.net/) allowing user-friendly DNA annotation with GENA-LM models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/nar/gkae1310

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 38

  1. Kim, 2023, Deciphering the multi-scale, quantitative cis-regulatory code, Mol. Cell, 83, 373, 10.1016/j.molcel.2022.12.032 · DOI · Paper page
  2. Whalen, 2022, Navigating the pitfalls of applying machine learning in genomics, Nat. Rev. Genet., 23, 169, 10.1038/s41576-021-00434-9 · DOI · Paper page
  3. Libbrecht, 2015, Machine learning applications in genetics and genomics, Nat. Rev. Genet., 16, 321, 10.1038/nrg3920 · DOI · Paper page
  4. Belokopytova, 2020, Quantitative prediction of enhancer–promoter interactions, Genome Res., 30, 72, 10.1101/gr.249367.119 · DOI · Paper page
  5. Sindeeva M., Chekanov N., Avetisian M., Shashkova T.I., Baranov N., Malkin E., Lapin A., Kardymon O., Fishman V. Cell type-specific interpretation of noncoding variants using deep learning-based methods. GigaScience. 2023; 12:giad015. · DOI
  6. Belokopytova, 2021, Predicting genome architecture: challenges and solutions, Front. Genet., 11, 617202, 10.3389/fgene.2020.617202 · DOI
  7. Pan, 2010, A survey on transfer learning, IEEE Trans. Knowl. Data Eng., 22, 1345, 10.1109/TKDE.2009.191 · DOI · Paper page
  8. Dai, 2015, Semi-supervised sequence learning, 3079
  9. Ji, 2021, DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome, Bioinformatics, 37, 2112, 10.1093/bioinformatics/btab083 · DOI · Paper page
  10. Vaswani, 2017, Attention is all you need, 5998
  11. Guo, 2019, Star-transformer, 10.18653/v1/N19-1133 · DOI · Paper page
  12. Ainslie, 2020, ETC: encoding long and structured data in transformers, 268, 10.18653/v1/2020.emnlp-main.19 · DOI · Paper page
  13. Zaheer, 2020, Big Bird: transformers for longer sequences, 17283
  14. Choromanski, 2021, Rethinking attention with performers
  15. Kang, 2023, The pan-genome and local adaptation of Arabidopsis thaliana, Nat. Commun., 14, 6259, 10.1038/s41467-023-42029-4 · DOI
  16. O’Donnell, 2023, Telomere-to-telomere assemblies of 142 strains characterize the genome structural landscape in Saccharomyces cerevisiae, Nat. Genet., 55, 1390, 10.1038/s41588-023-01459-y · DOI · Paper page
  17. Kim, 2024, Single-fly assemblies fill major phylogenomic gaps across the Drosophilidae Tree of Life, PLoS Biol., 22, e3002697, 10.1371/journal.pbio.3002697 · DOI
  18. Jaganathan, 2019, Predicting splicing from primary sequence with deep learning, Cell, 176, 535, 10.1016/j.cell.2018.12.015 · DOI · Paper page
  19. Zhou, 2015, Predicting effects of noncoding variants with deep learning-based sequence model, Nat. Methods, 12, 931, 10.1038/nmeth.3547 · DOI · Paper page
  20. Bogard, 2019, A deep neural network for predicting and engineering alternative polyadenylation, Cell, 178, 91, 10.1016/j.cell.2019.04.046 · DOI · Paper page
  21. Xiong, 2020, On layer normalization in the transformer architecture, 10524
  22. Loshchilov, 2019, Decoupled weight decay regularization
  23. Mukund, 2017, Axiomatic attribution for deep networks, 3319
  24. Grant, 2021, XSTREME: comprehensive motif analysis of biological sequence datasets, 10.1101/2021.09.02.458722 · DOI
  25. Kulakovskiy, 2018, HOCOMOCO: towards a complete collection of transcription factor binding models for human and mouse via large-scale ChIP-Seq analysis, Nucleic Acids Res., 46, D252, 10.1093/nar/gkx1106 · DOI · Paper page
  26. Rhee, 2012, ChIP-exo: a method to identify genomic location of DNA-binding proteins at near single nucleotide accuracy
  27. Kawana, 1995, Cooperative interaction of GATA-2 and AP1 regulates transcription of the endothelin-1 gene, Mol. Cell. Biol., 15, 4225, 10.1128/MCB.15.8.4225 · DOI · Paper page
  28. Menendez-Gonzalez, 2019, Inhibition of GATA2 restrains cell proliferation and enhances apoptosis and chemotherapy mediated apoptosis in human GATA2 overexpressing AML cells, Sci. Rep., 9, 12212, 10.1038/s41598-019-48589-0 · DOI
  29. Eferl, 2003, AP-1: a double-edged sword in tumorigenesis, Nat. Rev. Cancer, 3, 859, 10.1038/nrc1209 · DOI · Paper page
  30. Camellato, 2024, Synthetic reversed sequences reveal default genomic states, Nature, 628, 373, 10.1038/s41586-024-07128-2 · DOI
  31. Shmelev, 2024, GENA-Web - GENomic Annotations Web Inference using DNA language models, 10.1101/2024.04.26.591391 · DOI
  32. Zhang, 2022, Transformer for gene expression modeling (T-GEM): an interpretable deep learning model for gene expression-based phenotype predictions, Cancers, 14, 4763, 10.3390/cancers14194763 · DOI
  33. Le, 2022, Deep transformers and convolutional neural network in identifying DNA N6-methyladenine sites in cross-species genomes, Methods, 204, 199, 10.1016/j.ymeth.2021.12.004 · DOI
  34. Kabirova, 2023, Function and evolution of the loop extrusion machinery in animals, Int. J. Mol. Sci., 24, 5017, 10.3390/ijms24055017 · DOI
  35. Fishman, 2018, Interpreting chromosomal rearrangements in the context of 3-dimentional genome organization: a practical guide for medical genetics, Biochemistry (Mosc.), 83, 393, 10.1134/S0006297918040107 · DOI
  36. Shashkova, 2022, SEMA: antigen B-cell conformational epitope prediction using deep transfer learning, Front. Immunol., 13, 960985, 10.3389/fimmu.2022.960985 · DOI · Paper page
  37. Madani, 2023, Large language models generate functional protein sequences across diverse families, Nat. Biotechnol., 41, 1099, 10.1038/s41587-022-01618-2 · DOI · Paper page
  38. Wang, 2022, LM-GVP: an extensible sequence and structure informed deep learning framework for protein property prediction, Sci. Rep., 12, 6832, 10.1038/s41598-022-10775-y · DOI · Paper page

Cited by

See all 135

Metadata

Authors
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M.
Venue
Nucleic Acids Research
Published
2025-01-11
Volume
53
Issue
2
ISSN
  • 0305-1048 (ISSN-L)
  • 1362-4962
OpenAlex
W4406419344
Cited by
135
References
27
FWCI
15.34
Type
journal article
Language
en
Open access
gold
Funders
AIRI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.