Effective gene expression prediction from sequence by integrating long-range interactions

May combine two claims.

Enformer's predictions are not limited to conserved enhancers because it does not rely explicitly on nucleotide conservation statistics.

“by not relying explicitly on nucleotide conservation statistics, as the majority of tools do25, its predictions are not limited to conserved enhancers, which comprise a small proportion of all enhancers30”

• cites [25] Kircher, M., Xiong, C., Martin, B. & Schubach, M. Saturation mutagenesis of twenty disease-associated regulatory elements at single base-pair resolution. Nature 10, 3583 (2019). sourcePartly supported

  In Results, under “Computational tools predict expression effects poorly” (page not provided), the paper says conservation was informative for only a subset of its studied regulatory elements. It does not establish that conserved enhancers are a small proportion of all enhancers. Nor does it assess Enformer or whether its predictions extend to unconserved enhancers. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [30] Villar, D. et al. Enhancer evolution across 20 mammalian species. Cell 160, 554 (2015). sourceSupported

May combine two claims.

The performance of Enformer was comparable to, and in some cases even better than, the ABC score, a state-of-the-art method recently proposed specifically for enhancer prioritization.

“The performance of Enformer was comparable to, and in some cases even better than, the ABC score13, a state-of-the-art method recently proposed specifically for enhancer prioritization.”

• cites [13] Fulco, C. P. et al. Activity-by-contact model of enhancer-promoter regulation from thousands of CRISPR perturbations. Nat. Genet. 51, 1664–1669 (2019). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing and were also recently applied to model short DNA sequences.

“Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing (NLP)6,7 and were also recently applied to model short DNA sequences8.”

• cites [6] Vaswani, A. et al. Attention is all you need. in Advances in Neural Information Processing Systems 5998–6008 (2017).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [7] Brown, T. B. et al. Language models are few-shot learners. in Advances in Neural Information Processing Systems (2020). sourceSupported
• cites [8] Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics https://doi.org/10.1093/bioinformatics/btab083 (2021). sourceSupported

May combine two claims.

This performance increase is twice as large as the performance increase between Basenji1 and Basenji2 and closes one-third of the gap to experimental-level accuracy, estimated at 0.94.

“This performance increase is twice as large as the performance increase between Basenji1 (ref. 3) and Basenji2 (ref. 2) and closes one-third of the gap to experimental-level accuracy, estimated at 0.94 (Extended Data Fig. 2).”

• cites [3] Kelley, D. R. et al. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Res. 28, 739–750 (2018). sourceSupported
• cites [2] Kelley, D. R. Cross-species regulatory sequence activity prediction. PLoS Comput. Biol. 16, e1008050 (2020). sourceSupported

May combine two claims.

The ABC score relies on experimental data, such as a HiC-based interaction frequency and H3K27ac as input, whereas Enformer uses only DNA sequence as input and was never trained to explicitly locate enhancers.

“This is remarkable because the ABC score relies on experimental data, such as a HiC-based interaction frequency and H3K27ac as input (Fig. 2b, blue versus green, and Extended Data Fig. 7a), whereas Enformer uses only DNA sequence as input and was never trained to explicitly locate enhancers.”

• cites [13] Fulco, C. P. et al. Activity-by-contact model of enhancer-promoter regulation from thousands of CRISPR perturbations. Nat. Genet. 51, 1664–1669 (2019). sourcePartly supported

  The ABC model description states that the score uses H3K27ac ChIP-seq data and, in the described implementation, Hi-C contact frequency. Our copy does not address Enformer, so it cannot support the claims about Enformer's inputs or training. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Increasing the number of parameters improved model performance.

“We note that increasing the number of parameters improved model performance, consistent with recent advances in NLP7.”

• cites [7] Brown, T. B. et al. Language models are few-shot learners. in Advances in Neural Information Processing Systems (2020). sourceSupported

Classified as probably not a substantial claim.

Tissue-specific gene expression strongly depends on distal elements.

“The performance improvement was largest for CAGE, possibly because tissue-specific gene expression strongly depends on distal elements11.”

• cites [11] Heinz, S., Romanoski, C. E., Benner, C. & Glass, C. K. The selection and function of cell type-specific enhancers. Nat. Rev. Mol. Cell Biol. 16, 144–154 (2015). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The strength of the regularization was controlled by a single λ parameter, which was optimized using tenfold crossvalidation for each training set.

“The strength of the regularization was controlled by a single λ parameter, which was optimized using tenfold crossvalidation for each training set using the cv.glmnet function of the glmnet library in R.”

• cites [45] Klein, J. C. et al. A systematic evaluation of the design and context dependencies of massively parallel reporter assays. Nat. Methods 17, 1083–1091 (2020). sourcePartly supported

  The Online Methods, under “Model training” (page unavailable), says that a single λ parameter was optimized using tenfold cross-validation. It specifies optimization on the entire dataset, not for each training set, and does not identify the cv.glmnet function or glmnet library named in the citing sentence. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Enformer can make predictions for arbitrary sequences, which enables the synthetic design of enhancers that are optimized to exhibit cell-type specificity.

“it can make predictions for arbitrary sequences, which enables the synthetic design of enhancers that are optimized to exhibit cell-type specificity31”

• cites [31] Linder, J. et al. A generative neural network for maximizing fitness and diversity of synthetic DNA and protein sequences. Cell Syst. 11, 49–62 (2020). sourcePartly supported

  The Discussion (page unavailable) proposes that DENs could be used with gene-expression data to engineer cell-type-specific enhancers. The paper does not attribute arbitrary-sequence prediction or this design capability to Enformer, and presents cell-type-specific enhancer design as a possibility rather than a demonstrated result. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Finer resolution has potential benefits when data support it, but would extend the sequence length entering the quadratic complexity MHA and make the model engineering intractable on currently available hardware.

“Finer resolution has potential benefits when the data support it15, but would extend the sequence length entering the quadratic complexity MHA and make the model engineering intractable on currently available hardware.”

• cites [15] Avsec, Ž. et al. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nat. Genet. https://doi.org/10.1038/s41588-021-00782-6 (2021). sourceSupported

May combine two claims.

Enformer maximum Z-scores increased by greater than one s.d. for 228 CAGE datasets, relative to 46 decreased by one.

“Enformer maximum Z-scores increased by greater than one s.d. for 228 CAGE datasets, relative to 46 decreased by one.”

May combine two claims.

Basenji2 contribution scores were zero for sequences beyond 20 kb from the TSS due to the limited receptive field, thereby missing several enhancers.

“By contrast, the contribution scores of Basenji2 were zero for sequences beyond 20 kb from the TSS due to the limited receptive field, thereby missing several enhancers.”

May combine two claims.

Using Enformer predictions directly as scores, without training, performed comparably to the lasso-trained model and also outperformed the other submissions.

“Moreover, using the Enformer predictions directly as scores, without training, performed comparably to the lasso-trained model and also outperformed the other submissions.”

May combine two claims.

Enformer outperforms Basenji2 in gene expression prediction from sequence both across genes and across CAGE experiments for protein-coding genes.

“Enformer outperforms Basenji2 in gene expression prediction from sequence both across genes and across CAGE experiments for protein-coding genes.”

May combine two claims.

Enformer yielded greater predictive accuracy than ExPecto1 for both across-genes (0.850 versus 0.812 Spearman r) and across-tissues (0.451 versus 0.368 Spearman r) evaluation.

“Enformer also yielded greater predictive accuracy than ExPecto1, a model trained to predict gene expression levels measured by RNA-seq, for both across-genes (0.850 versus 0.812 Spearman r) and across-tissues (0.451 versus 0.368 Spearman r) evaluation (Extended Data Fig. 4).”

May combine two claims.

deltaSVM successfully predicted only one binding site but missed the other three, overall exhibiting 50% reduced Pearson and Spearman correlations to the measured effects relative to Enformer.

“By contrast, deltaSVM successfully predicted only one binding site but missed the other three, overall exhibiting 50% reduced Pearson and Spearman correlations to the measured effects relative to Enformer.”

May combine two claims.

Enformer contribution scores correlated with H3K27ac and highlighted not only local promoter regions, but also distal enhancers more than 20 kb away.

“We inspected the contribution scores of several genes and observed that they correlated with histone H3 acetylated at K27 (H3K27ac) and highlighted not only local promoter regions, but also distal enhancers more than 20 kb away (Fig. 2a and Supplementary Figs. 2 and 3).”

May combine two claims.

Enformer paid more attention to TAD boundaries than to random positions and less attention to regions on the opposite side of the boundary.

“From the perspective of the query position, Enformer paid more attention to TAD boundaries than to random positions (vertical red stripe, Fig. 2c) and less attention to regions on the opposite side of the boundary (off-diagonal blue blocks, Fig. 2c), consistent with the biological observation of reduced inter-TAD interactions.”

May combine two claims.

This led to a substantial performance increase in tissue- and cell-type-specific gene expression prediction correlation from 0.81 to 0.85, one-third of the way toward the experimental-level accuracy of 0.94 estimated from replicates.

“This led to a substantial performance increase in tissue- and cell-type-specific gene expression prediction correlation from 0.81 to 0.85, one-third of the way toward the experimental-level accuracy of 0.94 estimated from replicates.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Attention pooling initialization to 2 × 1 gave slightly better performance than random initialization or initialization with zeros.

“This initialization gave slightly better performance than did random initialization or initialization with zeros, representing average pooling.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Setting the maximum features considered per decision tree split to log2 of the total number of features greatly improved the computational efficiency.

“However, owing to the large number of features derived from the training datasets, setting the maximum features considered per decision tree split to log2 of the total number of features greatly improved the computational efficiency.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We selected the default hyperparameters of the scikit-learn 0.22 implementation after finding negligible accuracy gains from modifying them.

“We selected the default hyperparameters of the scikit-learn 0.22 implementation after finding negligible accuracy gains from modifying them44.”

• cites [44] Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011). sourceNot checked — no full text of the source

May combine two claims.

These results confirm that the Enformer architecture advances prediction accuracy for both a broad range of epigenetic marks and gene expression from DNA sequence.

“These results confirm that the Enformer architecture advances prediction accuracy for both a broad range of epigenetic marks and gene expression from DNA sequence.”

May combine two claims.

With a novel transformer architecture, the authors have made a significant improvement by greatly expanding the receptive field and increasing information flow between distal elements.

“With a novel transformer architecture, we have made a significant improvement by greatly expanding the receptive field and increasing the information flow between distal elements.”

May combine two claims.

The model not only better represents variants likely overlapping long-range enhancers, but also more effectively parses promoters and short-range enhancers.

“This improvement was consistent across all distances from the TSS (Fig. 3e), suggesting that the model not only better represents variants likely overlapping long-range enhancers (enabled by the larger receptive field), but also more effectively parses promoters and short-range enhancers.”

May combine two claims.

This improvement in predictive accuracy translated to improved models for two key problems of biological relevance: enhancer-promoter prediction and noncoding variant effect prediction.

“This improvement in predictive accuracy translated to improved models for two key problems of biological relevance: enhancer–promoter prediction and noncoding variant effect prediction.”

May combine two claims.

Overall, these results suggest that the model has learned about the role of tissue-specific enhancers and promoters, and also about insulator elements and their role in inhibiting information flow between genomic compartments.

“Overall, these results suggest that the model has not only learned about the role of tissue-specific enhancers and promoters, but also about insulator elements and their role in inhibiting information flow between genomic compartments.”

May combine two claims.

The model pays attention to enhancers and considers insulators when making gene expression predictions, suggesting that it has learned canonical distal regulation patterns.

“We observed that the model pays attention to enhancers and considers insulators when making gene expression predictions, suggesting that it has learned canonical distal regulation patterns.”

May combine two claims.

A limitation of the current approach is that we can model and predict only for cell types and assays in the training data and cannot generalize to new cell types or assays.

“A limitation of the current approach is that we can model and predict only for cell types and assays in the training data and cannot generalize to new cell types or assays.”

Classified as probably not a substantial claim.

We trained separate random forest classifiers for each tissue to distinguish causal from noncausal variants using eight-fold crossvalidation.

“To determine how informative different variant annotations are, we trained separate random forest classifiers for each tissue to distinguish causal from noncausal variants using eight-fold crossvalidation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Enformer architecture is similar to the state-of-the-art model Basenji2.

“The Enformer architecture is similar to the state-of-the-art model Basenji2 (ref. 2).”

• cites ref. 2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

We only evaluated the performance of our model on the test set once to generate Fig. 1 and did not use the test set during model development.

“We only evaluated the performance of our model on the test set once to generate Fig. 1 and did not use the test set during model development.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The optimal learning rate was discovered by grid search yielding the highest performance on the validation set.

“The optimal learning rate was discovered by grid search yielding the highest performance on the validation set.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We fit 100 iterations of stochastic crossvalidation shuffling and random forest fitting to delineate a low-variance estimate of model accuracy.

“We fit 100 iterations of stochastic crossvalidation shuffling and random forest fitting to delineate a low-variance estimate of model accuracy.”

Classified as probably not a substantial claim. May combine two claims.

The model was trained, evaluated, and tested on the same targets, genomic intervals, and using the same Poisson negative log-likelihood loss function as Basenji2.

“The model was trained, evaluated, and tested on the same targets, genomic intervals, and using the same Poisson negative log-likelihood loss function as Basenji2 (ref. 2).”

• cites ref. 2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

We used the validation set for hyperparameter selection and the test set for Basenji2 comparison.

“We used the validation set for hyperparameter selection and the test set for Basenji2 comparison.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All models were trained for 500,000 steps while only storing the model with the highest Spearman correlation of CAGE TSS gene expression across genes averaged across experiments computed on the validation set every 1,000 steps.

“All models were trained for 500,000 steps while only storing the model with the highest Spearman correlation of CAGE TSS gene expression across genes averaged across experiments computed on the validation set every 1,000 steps.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We used test-time augmentation during model evaluation: we averaged the predictions from 8 sequences randomly augmented the same way as during training (≤3 bp shifts and reverse-complementation).

“We used test–time augmentation during model evaluation: we averaged the predictions from 8 sequences randomly augmented the same way as during training (≤3 bp shifts and reverse-complementation).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The convolution tower preceding MHA in Enformer serves to embed nucleotide segments and contributes an inductive bias for adjacent nucleotides to function together in motifs.

“The convolution tower preceding MHA in the Enformer model serves to perform an analogous operation of embedding nucleotide segments and contributes a compelling inductive bias for adjacent nucleotides to function together in motifs.”