CoChat
Sign Up
Cited by 668Machine Learning in BioinformaticsGenomics and Phylogenetic StudiesGenomics and Rare Diseases

Nucleotide Transformer: building and evaluating robust foundation models for human genomics

Open access
Dalla-Torre, Hugo, Gonzalez, Liam, Mendoza-Revilla, Javier, Lopez Carranza, Nicolas, Grzywaczewski, Adam Henryk, Oteri, Francesco, Dallago, Christian, Trop, Evan +7 more·2025·Nature Methods
DOI 10.1038/s41592-024-02523-zPMC PMC11810778

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract Closing the gap between measurable genetic information and observable traits is a longstanding challenge in genomics. Yet, the prediction of molecular phenotypes from DNA sequences alone remains limited and inaccurate, often driven by the scarcity of annotated data and the inability to transfer learnings between prediction tasks. Here, we present an extensive study of foundation models pre-trained on DNA sequences, named the Nucleotide Transformer, integrating information from 3,202 diverse human genomes, as well as 850 genomes from a wide range of species, including model and non-model organisms. These transformer models yield transferable, context-specific representations of nucleotide sequences, which allow for accurate molecular phenotype prediction even in low-data settings. We show that the sequence representations alone match or outperform specialized methods on 12 of 18 prediction tasks, and up to 15 after fine-tuning. Despite no supervision, the transformer models learned to focus attention on key genomic elements, including those that regulate gene expression, such as enhancers. Lastly, we demonstrate that utilizing model representations can improve the prioritization of functional genetic variants. The training and application of foundational models in genomics explored in this study provide a widely applicable stepping stone to bridge the gap of accurate molecular phenotype prediction from DNA sequence.

AI snapshot

Written from the paper’s full text.

TL;DR

Nucleotide Transformer introduces and benchmarks DNA foundation models from 50 million to 2.5 billion parameters pretrained on the human reference genome, 3,202 human genomes, and 850 species genomes. Fine-tuned models match or exceed supervised baselines across many genomic tasks, learn regulatory elements without supervision, and prioritize variants, with optimized NT-v2 models achieving strong performance at much smaller sizes.

Key findings

  • Fine-tuned NT models matched (n = 6) or surpassed (n = 12) the 18 BPNet baseline models; probing alone matched 5 and exceeded 8 out of 18 tasks.
  • The Multispecies 2.5B model achieved the highest overall performance across the 18 tasks compared with DNABERT-2, HyenaDNA-32-kb, and Enformer, and matched or outperformed HyenaDNA in all 18 tasks (matched n = 7, outperformed n = 11).
  • On additional tasks, the Multispecies 2.5B model reached a top-k accuracy of 95% and precision-recall AUC of 0.98 for splice-site prediction; the NT-v2 500M 12-kb model improved this to 96% top-k accuracy and 0.98 precision-recall AUC.
  • The NT-v2 250M model attained the best benchmark performance (average MCC of 0.769) while being ten-times smaller than the 2.5B-parameter model; zero-shot variant scores reached AUCs of 0.7–0.8, with ClinVar AUC 0.80 for the Multispecies 2.5B model.

Methods

  • Encoder-only transformer language models trained with masked language modeling on six-mer tokens; NT-v1 used 6-kb context and NT-v2 used 12-kb context with rotary embeddings, swiGLU activations, and no MLP biases or dropout.
  • Pre-training datasets included the human reference genome, 3,202 high-coverage human genomes from the 1000 Genomes Project, and 850 species genomes; models ranged from 50 million to 2.5 billion parameters.
  • Evaluation used 18 curated genomic tasks with tenfold cross-validation, probing and parameter-efficient IA3 fine-tuning, comparisons with BPNet, DNABERT-2, HyenaDNA, and Enformer, plus additional DeepSEA, SpliceAI, and DeepSTARR tasks and zero-shot variant-effect scoring.

Limitations

  • The authors state that the models’ attention span remains limited; they note that Enformer suggested perception fields up to 200 kb may be needed for long-range dependencies, that standard transformers scale quadratically with sequence length, and that HyenaDNA performance deteriorated when its training perception field increased.
  • The authors state that supervised models with optimized architectures and extensive training on large datasets continue to demonstrate superior performance, though DNA language models remain competitive on several tasks.
  • The authors note that Enformer’s benchmark performance might be inflated due to potential data leakage from different original data splits; the text also makes evident that the 18-task benchmark datasets were subsampled to a maximum of 30,000 training and 3,000 validation/testing examples, which may limit generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41592-024-02523-z

Cites

95 of 96 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

14 of 30 cited sources checked · 5 supported · 1 flagged · accuracy not yet measured

  • Findingown data

    Crucial regulatory elements governing gene expression, such as enhancers and promoters, were consistently detected by all models across multiple heads and layers.

    “Crucial regulatory elements governing gene expression, such as enhancers and promoters41–43, were consistently detected by all models across multiple heads and layers.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [41] Lappalainen, T. et al. Transcriptome and genome sequencing uncovers functional variation in humans. Nature 501, 506–511 (2013). source
    • cites [42] Consortium, G. The gtex consortium atlas of genetic regulatory effects across human tissues. Science 369, 1318–1330 (2020). source
    • cites [43] Võsa, U. et al. Large-scale cis-and trans-eqtl analyses identify thousands of genetic loci and polygenic scores that regulate blood gene expression. Nat. Genetics 53, 1300–1310 (2021). source

      The Results section, “Local genetic effects on blood gene expression” (page unavailable), links promoter capture Hi-C contacts to genetic effects on gene expression, supporting the regulatory context. The paper does not itself discuss enhancers as gene-expression regulators; enhancers appear in a cited reference’s title. The statement about detection by models follows the citation and is not attributed to this source. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Interpretationown data

    Downstream model engineering can lead to performance improvements.

    “confirming that downstream model engineering can lead to performance improvements47.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [47] Stärk, H., Dallago, C., Heinzinger, M. & Rost, B. Light attention predicts protein location from the language of life. Bioinform. Adv. 1, vbab035 (2021). source
  • Interpretationreasoning

    It is probable that the models, without any supervision, have learned to distinguish relevant regulatory genomic features associated with gene expression and methylation variation.

    “it is probable that the models, without any supervision, have learned to distinguish relevant regulatory genomic features associated with gene expression and methylation variation.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [41] Lappalainen, T. et al. Transcriptome and genome sequencing uncovers functional variation in humans. Nature 501, 506–511 (2013). source

      The Properties of regulatory variants section reports that expression QTLs are enriched in regulatory genomic features, including transcription-factor peaks, promoters, and enhancers. It does not show that unsupervised models learned to distinguish those features or connect them to methylation variation; the Methods instead describe QTL mapping by linear regression. No page numbers are available in our copy. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [42] Consortium, G. The gtex consortium atlas of genetic regulatory effects across human tissues. Science 369, 1318–1330 (2020). source

      The Fine-mapping and Discussion sections discuss regulatory variants associated with gene expression, but neither reports that unsupervised models learned to distinguish genomic features. Searches for unsupervised learning and methylation variation found no passage addressing that claim; the available passages have no page numbers. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [43] Võsa, U. et al. Large-scale cis-and trans-eqtl analyses identify thousands of genetic loci and polygenic scores that regulate blood gene expression. Nat. Genetics 53, 1300–1310 (2021). source

      The Results section on local and distal gene expression describes eQTL meta-analyses, while Online Methods describes expression-data preprocessing and probe matching. Neither reports that unsupervised models learned to distinguish regulatory genomic features associated with both gene-expression and methylation variation (searched: unsupervised models, regulatory genomic features, methylation variation). (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Premisecitation

    Enformer has been originally trained in a supervised fashion to solve chromatin and gene expression tasks.

    “Note that Enformer has been originally trained in a supervised fashion to solve chromatin and gene expression tasks19.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [19] Avsec, Ž. et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nat. Methods 18, 1196–1203 (2021). source
  • Premisecitation

    The DeepSTARR enhancer activity dataset is composed of 484,052 DNA sequences of size 249 nucleotides, each measured for quantitative enhancer activity toward a developmental or a housekeeping promoter.

    “The dataset is composed of 484,052 DNA sequences of size 249 nucleotides, each measured for their quantitative enhancer activity toward a developmental or a housekeeping promoter.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites de Almeida et al.12 de Almeida, B. P., Reiter, F., Pagani, M. & Stark, A. DeepSTARR predicts enhancer activity from dna sequence and enables the de novo design of synthetic enhancers. Nat. Genet. 54, 613–624 (2022). source

      The Data preparation section states that the dataset contains 249-bp windows and 484,052 examples after augmentation. It also says reverse complements were added with the same output as their original sequences, so the paper does not support saying that each of the 484,052 sequences was separately measured. The DeepSTARR prediction section describes activities for both developmental and housekeeping promoters. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • May combine two claims. 46
  • Classified as probably not a substantial claim. 35

  • May combine two claims.

    Premisecitation

    DNABERT-1 can only handle a maximum input length of 512 bp and thus could not be used for most tasks.

    “We excluded DNABERT-1 from this comparison as it can only handle a maximum input length of 512 bp and thus could not be used for most tasks20.”
    • Also stated in: § Models · plain
      “We excluded DNABERT-1 from this comparison as it can only handle a maximum input length of 512 bp and thus could not be used for most tasks20.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [20] Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics 37, 2112–2120 (2021). source

      The Training section (§2.2) specifies a maximum input length of 512 for the tokenized sequence, not 512 bp. The Fine-tuning section (§2.2.3) says sequences longer than 512 are split into pieces so DNABERT-XL can process extra-long sequences; the paper also reports applications to several prediction tasks. This contradicts the claim that DNABERT could not be used for most tasks because of a 512-bp limit. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    In housekeeping and developmental enhancer prediction, the NT model slightly surpassed (1%) and obtained lower (4%) correlation values, respectively, compared to DeepSTARR.

    “in the case of housekeeping and developmental enhancer prediction, our model slightly surpassed (1%) and obtained lower (4%) correlation values, respectively (Fig. 2e), when compared to those of DeepSTARR12.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [12] de Almeida, B. P., Reiter, F., Pagani, M. & Stark, A. DeepSTARR predicts enhancer activity from dna sequence and enables the de novo design of synthetic enhancers. Nat. Genet. 54, 613–624 (2022). source

  • May combine two claims.

    Interpretationown data

    The limited separation of 3′ UTR regions from other elements suggests that the model has not fully learned to distinguish this type of element, or that many of these regions might be mis-annotated.

    “The limited separation of 3′ UTR regions from other elements suggests that the model has not fully learned to distinguish this type of element, or as previously suggested, that many of these regions might be mis-annotated37.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [37] Benegas, G., Batra, S. S. & Song, Y. S. DNA language models are powerful zero-shot predictors of non-coding variant effects. Proc. Natl Acad. Sci. USA 120, e2311219120 (2023). source

      The citation is attached to the suggestion that many 3′ UTR regions might be mis-annotated. In Results, “Unsupervised Clustering of Genomic Regions” (page unavailable), the paper suggests annotation errors for ncRNAs, not 3′ UTRs. Searches for 3′ UTR annotation and classification found no passage in our copy addressing that suggestion. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The BPNet architecture has been used widely in genomics and represents a very strong default architecture for modeling small-sized datasets from scratch through supervised learning.

    “The BPNet architecture has been used widely in genomics and represents a very strong default architecture for modeling small-sized datasets from scratch through supervised learning.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Avsec, Ž. et al. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nat. Genet. 53, 354–366 (2021). source

      The BPNet architecture and training sections describe a supervised model trained on genomic data, and the Discussion calls it a general framework that outperformed the methods evaluated. The paper does not establish that BPNet was already widely used in genomics or that it is a strong default for small-sized datasets trained from scratch; its reported training is not presented as a small-dataset evaluation. The supplied passages have section labels but no page numbers. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    The DeepSEA dataset is composed of 2.4 million sequences, each of size 1,000 nucleotides, and associated with 919 chromatin features, including 690 TF, 125 DNase and 104 histone features.

    “The dataset is composed of 2.4 million sequences, each of size 1,000 nucleotides, and associated with 919 chromatin features. These include 690 TF, 125 DNase and 104 histone features.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zhou et al.10 Zhou, J. & Troyanskaya, O. G. Predicting effects of noncoding variants with deep learning–based sequence model. Nat. Methods 12, 931–934 (2015). source

      Online Methods, “Data for training DeepSEA” (page not specified), says each sample is a 1,000-bp sequence paired with labels for 919 chromatin features. “Model design and training” gives the stated breakdown: 690 TF, 125 DNase and 104 histone features. The copy does not state that the dataset contains 2.4 million sequences; it describes 521,636,200 bp divided into 200-bp bins, with some chromosomes held out for testing. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    In comparison with the state-of-the-art enhancer activity DeepSTARR model, the NT model achieved similar performance for four TF motifs and demonstrated superior performance for the Dref motif.

    “In comparison with the state-of-the-art enhancer activity DeepSTARR model, our model achieved similar performance for four TF motifs and demonstrated superior performance for the Dref motif”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model captured constraints at different splicing junctions and identified a region in the middle of the second intron crucial for the splicing of this exon.

    “The model not only captured constraints at different splicing junctions but also identified a region in the middle of the second intron crucial for the splicing of this exon.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Fine-tuned NT models either matched (n = 6) or surpassed (n = 12) out of the 18 baseline models.

    “our fine-tuned models either matched (n = 6) or surpassed (n = 12) out of the 18 baseline models”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Parameter-efficient fine-tuning allowed for faster fine-tuning on a single GPU, reduced storage needs by 1,000-fold over all fine-tuning parameters, while still delivering comparable performance.

    “This approach allowed for faster fine-tuning on a single GPU, reduced storage needs by 1,000-fold over all fine-tuning parameters, while still delivering comparable performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Fine-tuning exhibited a smaller variance in performance, enhancing the robustness of the approach.

    “Additionally, fine-tuning exhibited a smaller variance in performance, enhancing the robustness of the approach.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The 500-M-sized models and those trained on less-diverse sequences exhibited lower separation among genomic regions.

    “In particular, the 500-M-sized models and those trained on less-diverse sequences, exhibited lower separation among genomic regions”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Enformer achieved the best performance on enhancer prediction and some chromatin tasks.

    “Still, Enformer achieved the best performance on enhancer prediction and some chromatin tasks”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Multispecies 2.5B model, the strongest separation at layer 1 was observed between intergenic and genic regions, followed by 5′ UTR regions on layer 5, and a separation between most regions on layer 21.

    “In the case of the Multispecies 2.5B model, the strongest separation at layer 1 was observed between intergenic and genic regions, followed by 5′ UTR regions on layer 5, and a separation between most regions on layer 21”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The number of significant attention heads across layers varied markedly across models, with the highest number of significant attention heads observed for the Multispecies 2.5B model for introns (117 out of 640 heads), exons (n = 72) and TF binding sites (n = 74).

    “The number of significant attention heads across layers varied markedly across models, with the highest number of significant attention heads observed for the Multispecies 2.5B model for introns (117 out of 640 heads), exons (n = 72) and transcription factor (TF) binding sites (n = 74)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There was no significant improvement in chromatin and splicing predictions, and only a modest 3% enhancement in enhancer activity predictions, when comparing parameter-efficient fine-tuning to full-model fine-tuning.

    “we observed no significant improvement in chromatin and splicing predictions, and only a modest 3% enhancement in enhancer activity predictions”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    NT models matched the baseline BPNet models in 5 tasks and exceeded them in 8 out of the 18 tasks through probing alone.

    “NT models matched the baseline BPNet models in 5 tasks and exceeded them in 8 out of the 18 tasks through probing alone”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The highest model performance is never achieved by using embeddings from the final layer.

    “We also noted that the highest model performance is never achieved by using embeddings from the final layer, as shown in earlier work5.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Elnaggar, A. et al. ProtTrans: towards cracking the language of life’s code through self-supervised deep learning and high performance computing. IEEE Trans. Pattern Anal. Mach. Intell. 44, 7112–7127 (2022). source

  • May combine two claims.

    Findingown data

    For nucleotide-level splice site prediction, the adapted NT model achieved a top-k accuracy of 95% and a precision-recall AUC of 0.98.

    “we adapted the NT model to deliver nucleotide-level splice site predictions and achieved a top-k accuracy of 95% and a precision-recall AUC of 0.98”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Regarding enhancers, the maximum attention percentages were highest for the largest models, with the 1000G 2.5B model achieving nearly 100% attention.

    “Regarding enhancers, the maximum attention percentages were highest for the largest models, with the 1000G 2.5B model, for instance, achieving nearly 100% attention”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The IA3 fine-tuning technique yielded superior results compared to extensive probing while using fewer compute resources.

    “In comparison to the extensive probing exercise, this technique yielded superior results while using fewer compute resources”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The best-performing models for prioritizing molecular phenotypes (eQTLs and meQTLs) were those trained on human sequences, whereas the best-performing model for prioritizing pathogenic variants was based on multispecies sequences.

    “The best-performing models for prioritizing molecular phenotypes (eQTLs and meQTLs) were those trained on human sequences, whereas the best-performing model for prioritizing pathogenic variants was based on multispecies sequences.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The NT-v2 250M-parameter model attained the best performance on the benchmark, with an average MCC of 0.769, while being ten-times smaller than the 2.5B-parameter model.

    “Of particular interest is the NT-v2 250M-parameter model, which attained the best performance on our benchmark (average MCC of 0.769) while being ten-times smaller than the 2.5B-parameter model (Fig. 5c).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Transformer models can match and even outperform other methods for predicting variant effects and deleteriousness.

    “We have demonstrated that transformer models can match and even outperform other methods for predicting variant effects and deleteriousness.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The probing strategy resulted in 760,000 downstream models trained.

    “This probing strategy resulted in 760,000 downstream models trained”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    The fully fine-tuned NT 2.5B Multispecies model was the model with the highest test set performance.

    “as it was the model with the highest test set performance”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both intra-species and inter-species variability significantly influence accuracy across genomic prediction tasks.

    “Our results, based on distinct genomic prediction tasks, demonstrate that both intra-species (when training on multiple genomes of a single species) and inter-species (on genomes across different species) variability significantly influences accuracy across tasks (Figs. 1c and 2a).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The NT-v2 250M- and 500M-parameter models achieved improved performance over the 2.5B-parameter multispecies model while maintaining a significantly leaner parameter count and doubling the perception field.

    “Indeed, the NT-v2 250M- and 500M-parameter model managed to achieve improved performance over the 2.5B-parameter multispecies model while maintaining a significantly leaner parameter count and doubling the perception field.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The models have acquired the ability to reconstruct genetic variants and to recognize key regulatory genomic elements.

    “we have demonstrated that the models have acquired the ability not only to reconstruct genetic variants (Supplementary Note) but also to recognize key regulatory genomic elements.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    A model was successfully constructed that achieves the performance of the previous best model with only one-tenth the number of parameters and while doubling the perception field size.

    “We successfully constructed a model that achieves the performance of our previous best model with only one-tenth the number of parameters and while doubling the perception field size.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The IA3 fine-tuning approach requires only 0.1% of the total number of parameters, allowing even the largest models to be fine-tuned in under 15 min on a single GPU.

    “Notably, this fine-tuning approach only requires 0.1% of the total number of parameters, allowing even our largest models to be fine-tuned in under 15 min on a single GPU.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The NT-v2 500M 12-kb model improved performance by 1% over NT 6-kb models on the SpliceAI splicing task, reaching a top-k accuracy of 96% and a precision-recall AUC of 0.98.

    “Compared to the NT 6-kb models, our NT-v2 500M 12-kb model improved performance by 1% to a top-k accuracy of 96% and a precision-recall AUC of 0.98 (Fig. 5e).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    After filters, 80,590 eQTL, 11,734 meQTL, 70,224 ClinVar and 14,626 HGMD genetic variants were retained.

    “After these filters, we retained a total of 80,590, 11,734, 70,224 and 14,626 genetic variants for the eQTLs, meQTLs, ClinVar and HGMD SNP datasets, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The promoter sequence dataset resulted in 29,598 promoter regions, 3,065 of which were TATA-box promoters.

    “This resulted in 29,598 promoter regions, 3,065 of which were TATA-box promoters (using the motif annotation at https://epd.expasy.org/ftp/epdnew/H_sapiens/006/db/promoter_motifs.txt).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Eukaryotic Promoter Database30

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    After keeping only SNPs and filtering out variants annotated to more than one consequence, a final dataset composed of 920 genetic variants per class was obtained.

    “After keeping only SNPs and filtering out variants annotated to more than one consequence (for example those annotated as stop-gained and splice variants), we obtained a final dataset composed of 920 genetic variants per class.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The 18 genomic datasets were processed into a standardized format to facilitate experimentation and ensure reproducibility in the assessment of large LM performance.

    “These datasets were processed into a standardized format to facilitate experimentation and ensure reproducibility in the assessment of large LM performance (Methods).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    A tenfold cross-validation strategy was implemented to ensure a fair and accurate comparison between different models.

    “To ensure a fair and accurate comparison between different models, we implemented a tenfold cross-validation strategy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The extensive benchmarks and results demonstrate the flexibility and performance of NT as a general approach to tackle many diverse genomics tasks with high accuracy.

    “Altogether, our extensive benchmarks and results demonstrate the flexibility and performance of NT as a general approach to tackle many diverse genomics tasks with high accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The NT models have acquired the ability to recover gene structure and functional properties of genomic sequences and integrate them directly into its attention mechanism.

    “Collectively, these results illustrate how the NT models have acquired the ability to recover gene structure and functional properties of genomic sequences and integrate them directly into its attention mechanism.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The NT models have learned to detect known genomic elements within their embeddings in an unsupervised manner, which can be harnessed for efficient downstream genomics task predictions.

    “This demonstrated that the NT models have learned to detect known genomic elements within their embeddings in an unsupervised manner, which can be harnessed for efficient downstream genomics task predictions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The encoded information should prove helpful in evaluating the significance of genetic variants.

    “This encoded information should prove helpful in evaluating the significance of genetic variants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premiseown data

    The Human reference dataset reached a total of 3.2 billion nucleotides.

    “The Human reference dataset was constructed by considering all autosomal and sex chromosomes sequences from the reference assembly GRCh38/hg38 (https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.26) and reached a total of 3.2 billion nucleotides.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The total number of heads of each model was used as a Bonferroni correction to the significance level (α) of 0.05.

    “The total number of heads of each model was used as a Bonferroni correction to the significance level (α) of 0.05.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Binary or multi-class MCC was used as the metric for consistent performance evaluation across tasks because it is robust to uneven label ratios.

    “For a consistent performance evaluation across tasks, we used the binary or multi-class MCC as the metric as it is robust to uneven label ratios.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    For functional variant prioritization, training and test sets were divided by chromosomes and strictly kept as nonoverlapping.

    “We employed a similar strategy to the one previously described, with the primary difference being that the training and test sets were divided by chromosomes and strictly kept as nonoverlapping.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A tenfold cross-validation scheme was used to evaluate each model, with nine parts for training and one-tenth reserved for validation and selecting the final checkpoint.

    “We used a tenfold cross-validation scheme to evaluate each model, where the training set is split into ten folds, and each time we used nine of those parts for training and reserved one-tenth for validation and selecting the final checkpoint.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    For final comparison across models, the mean MCC across the three task categories (chromatin profiles, regulatory elements and splicing) was calculated, using the median MCC across tasks for each category.

    “For a final comparison across models, we calculated for each model the mean MCC across the three different categories of task (chromatin profiles, regulatory elements and splicing), where for each category we used the median MCC across tasks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Model training and performance evaluation were performed on different sets of chromosomes from the human genome; sequences from chromosomes 20 and 21 were used for the test and the remainder for training.

    “Model training and performance evaluation were performed on different sets of chromosomes from the human genome. Namely, sequences from chromosomes 20 and 21 were used for the test and the remainder were used for training the different models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The median performance across the ten folds was used as the final performance metric of each model on a given task.

    “We used the median performance across the ten folds as the final performance metric of each model on a given task.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Attention heads with P values below the Bonferroni-corrected significance level were considered significant.

    “Attention heads with P values below the Bonferroni-corrected significance level were considered to be significant.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    A tenfold validation scheme was used, splitting the training dataset ten times into training and validation sets with 90% and 10% of the initial set.

    “We used a tenfold validation scheme, where the training dataset was split ten times in a training and validation set, that contain different shuffles with 90% and 10% of the initial set.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A feature was considered captured by an attention head if pα(f) was significantly greater than the natural occurring frequency of the feature within the dataset.

    “We considered that a feature was captured by an attention head if the quantity pα(f) was significantly greater than the natural occurring frequency of the feature within the dataset (Supplementary Table 10).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    The NT-v2 context length of 12 kb is 24× and 4× larger than the respective average context lengths of DNABERT-1 (500 bp) and DNABERT-2 (3 kb).

    “To put this in perspective, this length is 24× and 4× larger than the respective average context lengths of DNABERT-1 (500 bp) and DNABERT-2 (3 kb).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The results affirmed the utility and effectiveness of both NT v1 and v2 models for a wide range of genomics tasks, requiring minimal modifications and computing power while achieving high accuracy.

    “In summary, these results affirmed the utility and effectiveness of both NT v1 and v2 models for a wide range of genomics tasks, requiring minimal modifications and computing power while achieving high accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Perplexity measures how well a model can reconstruct masked positions and is a more-refined measure than accuracy because it accounts for magnitude; lower perplexity suggests better reconstruction ability and thus better performance.

    “Perplexity measures how well a model can reconstruct masked positions and is a more-refined measure than accuracy as it does also account for magnitude. In contrast to accuracy, lower perplexity suggests better reconstruction ability, thus better performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rives4

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The advanced NT-v2 models exhibit improved downstream performance and are suitable for execution and fine-tuning on economical hardware.

    “These advanced models not only exhibit improved downstream performance but also offer the advantage of being suitable for execution and fine-tuning on economical hardware.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    It is possible that the multispecies model leveraged sequence variation across species to learn about the degree of conservation across sites.

    “it is possible that the multispecies model leveraged sequence variation across species to learn about the degree of conservation across sites.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The most severely pathogenic variants tend to impact gene function due to amino acid changes.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    In the splice site prediction dataset, annotations were filtered to exclude level-3 transcripts (automated annotation), so that all training data were annotated by a human.

    “Annotations were filtered to exclude level-3 transcripts (automated annotation), so that all training data were annotated by a human.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A two-proportion z-test was conducted with the null hypothesis as the natural frequency of the feature and the alternate hypothesis as pα(f).

    “To validate this, we conducted a two-proportion z-test with the null hypothesis as the natural frequency of the feature, and the alternate hypothesis as pα(f).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To ensure each embedding is associated with a six-mer, sequences were cut so that their length was divisible by 6.

    “To ensure that each embedding is associated with a six-mer, the sequences were cut so that their length was divisible by 6.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    For benchmarking, the provided Enformer model torso was reused as a pretrained model, which is not the intended and recommended use of the original paper.

    “For the sake of benchmarking, we reused the provided model torso as a pretrained model for our benchmark, which is not the intended and recommended use of the original paper”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premiseown data

    The Multispecies dataset collection was downsampled to 850 species, whose genomes add up to 174 billion nucleotides.

    “The resulting collection of genomes was downsampled to a total of 850 species, whose genomes add up to 174 billion nucleotides.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premiseown data

    The 1000G dataset contained 3,202 high-coverage human genomes from 27 geographically structured populations, making up a total of 20.5 trillion nucleotides.

    “The dataset contained 3,202 high-coverage human genomes, originating from 27 geographically structured populations of African, American, East Asian and European ancestry as detailed in Supplementary Table 2, making up a total of 20.5 trillion nucleotides.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Full fine-tuning was performed only for HyenaDNA due to incompatibility of the parameter-efficient fine-tuning approach with the model architecture.

    “Only for HyenaDNA we performed full fine-tuning due to the incompatibility of our parameter-efficient fine-tuning approach with the model architecture.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    All sequences with Ns were removed from both training and test set in the SpliceAI benchmark, which represented a negligible portion of the data.

    “Furthermore, all sequences with Ns were removed from both training and test set, which represented a negligible portion of the data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premiseown data

    The 1000G phased data corresponded to a total of 125 million mutations, of which 111 million are SNPs and 14 million are indels.

    “To allow haplotype reconstruction in the FASTA format from the VCF files, we considered the phased version of the data, which corresponded to a total of 125 million mutations, 111 million and 14 million of which are single-nucleotide polymorphisms (SNPs) and indels, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Transformer models trained on diverse species have learned to capture genomic features that likely have functional importance across species, thus enabling better generalization in various human-based prediction tasks.

    “This suggests that transformer models trained on diverse species have learned to capture genomic features that likely have functional importance across species, thus enabling better generalization in various human-based prediction tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The SpliceAI training dataset was constructed using GENCODE V24lift37 annotations and RNA-seq data from the GTEx cohort, focusing solely on splice site annotations from the principal transcript.

    “In brief, this training dataset was constructed using GENCODE V24lift37 annotations and RNA-seq data from the GTEx cohort, focusing solely on splice site annotations from the principal transcript.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites SpliceAI36

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites GENCODE V24lift37

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites GTEx

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    For enhancer activity prediction, the regression task was treated as a multilabel classification problem following methodology used elsewhere.

    “Following the methodology used elsewhere3, we chose to treat this regression task as a multilabel classification problem.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites elsewhere3

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The SpliceAI training dataset comprised annotations from genes on chromosomes 2, 4, 6, 8, 10-22, X and Y, while annotations from genes on the remaining chromosomes, which were not paralogs, constituted the test dataset.

    “The training dataset comprised annotations from genes located on chromosomes 2, 4, 6, 8 and 10–22, as well as chromosomes X and Y, while annotations from genes on the remaining chromosomes, which were not paralogs, constituted the test dataset.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites SpliceAI36

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For functional variant prioritization, negative variants were constructed from 1000G Project SNPs with minor allele frequency >5% that did not overlap with any variant in the tested dataset and were within 100 kb of the associated variants, resulting in four balanced datasets.

    “For each of these four datasets, we then constructed a set of negative variants based on SNPs from the 1000G Project with a minor allele frequency >5% that did not overlap with any variant reported in the dataset tested and that were within 100 kb of the associated variants, resulting in four balanced datasets.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites 1000G Project

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    In chromatin profile prediction, losses associated with positive samples were upscaled by a factor of 8 because each label is highly unbalanced and composed mostly of negative samples.

    “As each label is highly unbalanced and is composed mostly of negative samples, the losses associated with positive samples are upscaled by a factor of 8.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Each sequence produced by the SpliceAI pipeline had a length of 15,000 bp, with the central 5,000 bp containing the sites to be predicted.

    “Each sequence produced by this pipeline had a length of 15,000 bp, with the central 5,000 bp containing the sites to be predicted.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites SpliceAI36

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

72 of 72

  • The paper relies on this without support.

    The most severely pathogenic variants tend to impact gene function due to amino acid changes.

    “Given that the most severely pathogenic variants tend to impact gene function due to amino acid changes”

  • The paper relies on this without support.

    The 18 downstream genomic prediction tasks are representative of a wide range of genomics tasks.

    “In summary, these results affirmed the utility and effectiveness of both NT v1 and v2 models for a wide range of genomics tasks”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This includes constructing large foundation models for genomics, clinical images, and clinical text learning.

    “This includes constructing large foundation models for genomics [186], clinical images [187], and clinical text learning [188].”

    From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [186]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting training durations for the NT-v2 250M model

    Minor

    The methods say the NT-v2 250M model was trained for both 300 billion and up to 1 trillion tokens. The stated 800-billion-token checkpoint also cannot come from a run stopped at 300 billion tokens.

  • Enhancer-attention result points to an embeddings panel

    Minor

    The text cites Fig. 3a for attention to enhancers, but the caption identifies Fig. 3a as an embeddings projection. It identifies Fig. 3d as the panel showing attention percentages for enhancers.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Li, F.-Z., Amini, A. P., Yang, K. K. & Lu, A. X. Pretrained protein language model transfer learning: is the final layer representation what we want? in Machine Learning for Structural Biology Workshop (NeurIPS, 2022).

Cited by

Sorted by citations

Related

Authors
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., Richard, G., Skwark, M., Beguir, K., Lopez, M., & Pierrot, T.
Affiliations
Nvidia (United States) (US)
Corresponding author
  • Lopez, Marie
  • Pierrot, Thomas
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2024-11-28
Volume
22
Issue
2
Pages
287-297
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/s41592-024-02523-z
PMC
PMC11810778
OpenAlex
W4404821554
Semantic Scholar
aa0f6f7393ee9663c75832d4f253d380e8ea0f6e
Cited by
656
References
74
Citations by year
2026: 313 · 2025: 247 · 2024: 85 · 2023: 11
FWCI
75.87
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion, cc-by-nc-nd)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
Keywords
Nucleotide Transformer, transformer models, low-data settings, cross-species transfer learning, variant prioritization, human genome
MeSH terms
Algorithms, Base Sequence, Humans, Models, Genetic, Phenotype, Software, Genome, Human, Sequence Analysis, DNA, Computational Biology, Genomics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review