Transfer learning enables predictions in network biology

Genes known to be highly context-dependent, such as NOTCH receptors, showed more variability in their embeddings across variable cell types compared to the known housekeeping gene GAPDH.

“Furthermore, genes known to be highly context-dependent, such as NOTCH receptors, showed more variability in their embeddings across variable cell types14 compared to the known housekeeping gene GAPDH (Extended Data Fig. 3).”

• cites [14] Li, Y. et al. Single-cell transcriptome analysis reveals dynamic cell populations and differential gene expression patterns in control and aneurysmal human aortic tissue. Circulation 142, 1374–1388 (2020). sourceCould not verify

  The Results sections describe cell-type-specific gene expression in human aortic tissue, but the passages found do not discuss NOTCH receptors as context-dependent genes. Searches for NOTCH, GAPDH, and embedding variability found no passage addressing the cited point. The embedding comparison is attributed to the citing paper’s Extended Data Fig. 3, not clearly to this source. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Mapping the gene regulatory networks that drive disease progression enables screening for molecules that correct the network by normalizing core regulatory elements, rather than targeting peripheral downstream effectors that may not be disease modifying.

“Mapping the gene regulatory networks that drive disease progression enables screening for molecules that correct the network by normalizing core regulatory elements, rather than targeting peripheral downstream effectors that may not be disease modifying4,5.”

• cites [4] Theodoris, C. V. et al. Human disease modeling reveals integrated transcriptional and epigenetic mechanisms of NOTCH1 haploinsufficiency. Cell 160, 1072–1086 (2015). sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [5] Theodoris, C. V. et al. Network-based screen in iPSC-derived cells reveals therapeutic candidate for heart valve disease. Science 371, eabd0724 (2021). sourceSupported

Geneformer cell embeddings clustered primarily by cell type and phenotype as opposed to individual patient.

“Using a publicly available aortic aneurysm dataset14 as a test case, we found that while the original data was impacted by inter-patient variability, Geneformer cell embeddings clustered primarily by cell type and phenotype as opposed to individual patient (Extended Data Fig. 4a).”

• cites [14] Li, Y. et al. Single-cell transcriptome analysis reveals dynamic cell populations and differential gene expression patterns in control and aneurysmal human aortic tissue. Circulation 142, 1374–1388 (2020). sourceSupported

Embeddings of genes in iPSC-derived myogenic cells showed similar context awareness with in silico differentiation via MYOD.

“Embeddings of genes in iPSC-derived myogenic cells16 showed similar context awareness with in silico differentiation via MYOD (Extended Data Fig. 2d–e).”

• cites [16] Guo, D. et al. iMyoblasts for ex vivo and in vivo investigations of human myogenesis and disease modeling. eLife 11, e70341 (2022). sourceSupported

In silico reprogramming fibroblasts by artificially adding OCT4, SOX2, KLF4, and MYC to the front of their rank value encodings significantly shifted the remaining genes in the transcriptome's embedding towards the iPSC state.

“When we in silico reprogrammed fibroblasts15 by artificially adding OCT4, SOX2, KLF4, and MYC to the front of their rank value encodings, the remaining genes in the transcriptome significantly shifted their embedding towards the iPSC state (Extended Data Fig. 2b–c).”

• cites [15] Xing, Q. R. et al. Diversification of reprogramming trajectories revealed by parallel single-cell transcriptome and chromatin accessibility sequencing. Sci. Adv. 6, 463–474 (2020). sourceSupported

In silico deletion of GATA4 had a significantly higher effect on genes known to be most significantly dysregulated by GATA4 variants in a previously reported iPSC disease model of GATA4-related heart defects.

“In silico deletion of GATA4 had a significantly higher effect on genes known to be most significantly dysregulated by GATA4 variants in a previously reported iPSC disease model of GATA4-related heart defects33 (Extended Data Fig. 9e).”

• cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). sourceSupported

Direct GATA4 targets (as defined by ChIP-seq) were significantly more impacted by in silico deletion of GATA4 in fetal cardiomyocytes compared to indirect targets.

“Notably, direct GATA4 targets (as defined by ChIP-seq33) were significantly more impacted by in silico deletion of GATA4 in fetal cardiomyocytes compared to indirect targets (Fig. 5a).”

• cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). sourceSupported

In silico deletion of GATA4 or TBX5 alone had a significantly more deleterious effect on their known co-bound targets compared to housekeeping genes.

“Indeed, in silico deletion of GATA4 or TBX5 alone had a significantly more deleterious effect on their known co-bound targets33 compared to housekeeping genes.”

• cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). sourceSupported

20% of attention heads significantly attended transcription factors more than other genes.

“When examining the attention weights in aortic ECs14, we found that 20% of attention heads significantly attended transcription factors more than other genes, indicating that specific attention heads learned, in an entirely self-supervised manner, the relative importance of transcription factors in distinguishing cell states (Fig. 4e).”

• cites [14] Li, Y. et al. Single-cell transcriptome analysis reveals dynamic cell populations and differential gene expression patterns in control and aneurysmal human aortic tissue. Circulation 142, 1374–1388 (2020). sourceSupported

In silico deletion of TBX5 in fetal cardiomyocytes more significantly impacted its direct targets (as defined by ChIP-seq) compared to indirect targets and housekeeping genes.

“Analogously, in silico deletion of TBX5, another known congenital heart disease gene, in fetal cardiomyocytes23 more significantly impacted its direct targets (as defined by ChIP-seq34) compared to indirect targets and housekeeping genes (Extended Data Fig. 9f).”

• cites [23] Cao, J. et al. A human cell atlas of fetal gene expression. Science 370, 808 (2020). sourceSupported
• cites [34] Kathiriya, I. S. et al. Modeling human TBX5 haploinsufficiency predicts regulatory networks for congenital heart disease. Dev. Cell 56, 292–309 (2021). sourcePartly supported

  The Results sections on cardiomyocyte function and transcriptional responses report TBX5 loss-of-function in iPSC-derived cardiomyocytes and an association between TBX5-sensitive genes and TBX5 binding sites. They do not report an in silico deletion in fetal cardiomyocytes. The direct-versus-indirect-target comparison is pointed to the citing paper’s Extended Data Fig. 9f, rather than clearly attributed to this source. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

By fine-tuning Geneformer using solely transcriptional data with only 56 labeled loci in ~15,000 ESCs, the model could predict the results of more recent studies that included genome-wide profiling of bivalent domains.

“Thus, by fine-tuning Geneformer using solely transcriptional data with only 56 labeled loci in ~15,000 ESCs, the model could predict the results of more recent studies30 that included genome-wide profiling of bivalent domains.”

• cites [30] Pan, G. et al. Whole-genome analysis of histone H3 lysine 4 and lysine 27 methylation in human embryonic stem cells. Cell Stem Cell 1, 299–312 (2007). sourceSupported

Chen et al. previously systematically integrated thousands of transcription factor binding and histone modification profiles assayed by ChIP-seq with thousands of gene expression profiles to identify two classes of transcription factors with distinct ranges of regulatory influence.

“Chen et al. previously systematically integrated thousands of transcription factor binding and histone modification profiles assayed by chromatin immunoprecipitation sequencing (ChIP-seq) with thousands of gene expression profiles to identify two classes of transcription factors with distinct ranges of regulatory influence31.”

• cites [31] Chen, C. H. et al. Determinants of transcription factor regulatory range. Nat. Commun. 11, 2472 (2020). sourceSupported

In silico deletion of genes linked by a prior GWAS to cardiac MRI traits relevant to cardiac disease had a larger effect compared to the control set.

“In silico deletion of genes linked by a prior genome-wide association study24 (GWAS) to cardiac magnetic resonance imaging (MRI) traits relevant to cardiac disease also had a larger effect compared to the control set (Extended Data Fig. 7b).”

• cites [24] Pirruccello, J. P. et al. Analysis of cardiac magnetic resonance imaging in 36,000 individuals yields genetic insights into dilated cardiomyopathy. Nat. Commun. 11, 2254 (2020). sourceSupported

Fine-tuned Geneformer to distinguish cardiomyocytes from non-failing hearts (n=9) or hearts affected by hypertrophic (n=11) or dilated (n=9) cardiomyopathy with an overall out-of-sample accuracy of 90%.

“First, we fine-tuned Geneformer to distinguish cardiomyocytes35 from non-failing hearts (n=9) or hearts affected by hypertrophic (n=11) or dilated (n=9) cardiomyopathy with an overall out-of-sample accuracy of 90% (Fig. 6b, Extended Data Fig. 10a).”

• cites [35] Chaffin, M. et al. Single-nucleus profiling of human dilated and hypertrophic cardiomyopathy. Nature 608, 174–180 (2022). sourceSupported

Geneformer was able to distinguish central versus peripheral factors in the N1-dependent network with fine-tuning on 884 ECs from healthy versus dilated aortas to a better degree than the alternative methods' predictions trained on the larger dataset of ~30,000 ECs.

“Interestingly, Geneformer was able to distinguish central versus peripheral factors in the N1-dependent network with fine-tuning on this very minimal data to a better degree than the alternative methods’ predictions trained on the larger dataset of ~30,000 ECs32, demonstrating the strength of pretraining in enabling predictions from increasingly limited data (Fig. 4d, Extended Data Fig. 9b).”

• cites [32] Litviňuková, M. et al. Cells of the adult human heart. Nature 588, 455–472 (2020). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

In silico deletion of TBX5 was significantly more deleterious to previously reported TBX5 direct targets.

“In silico deletion of TBX5 was significantly more deleterious to previously reported TBX5 direct targets34”

• cites [34] Kathiriya, I. S. et al. Modeling human TBX5 haploinsufficiency predicts regulatory networks for congenital heart disease. Dev. Cell 56, 292–309 (2021). sourcePartly supported

  The Results section, “Discrete transcriptional responses to reduced TBX5 dosage in cardiomyocytes” (page unavailable), reports that TBX5 occupancy was associated with dosage-vulnerable genes and that 61 of 85 stepwise dose-dependent genes were near TBX5 binding sites. It does not report an in silico TBX5 deletion or a statistically significant comparison showing greater harm to previously reported direct targets. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

As cells pass through each layer of the Geneformer model fine-tuned to distinguish indicated cell types using only Drop-seq data, the model successively extrudes them from each other to derive separable embeddings that distinguish the cell types.

“As the cells pass through each layer, the model successively extrudes them from each other to derive separable embeddings that distinguish the cell types.”

• cites [11] Selewa, A. et al. Systematic comparison of high-throughput single-cell and single-nucleus transcriptomes during cardiomyocyte differentiation. Sci. Rep. 10, 1535 (2020). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Cells which do not progress to the iPSC state bifurcate into an alternative fate compared to cells that progress to the iPSC state after the day 12 stage.

“Cell embeddings suggested that cells which do not progress to the iPSC state bifurcate into an alternative fate compared to cells that progress to the iPSC state after the day 12 stage.”

• cites [15] Xing, Q. R. et al. Diversification of reprogramming trajectories revealed by parallel single-cell transcriptome and chromatin accessibility sequencing. Sci. Adv. 6, 463–474 (2020). sourcePartly supported

  The Results section on pseudotemporal trajectories reports branching paths and distinguishes a non-reprogrammed state 8 from a successfully reprogrammed state 9. It also places unsuccessful and successful subgroups as early as day 8, so the copy does not establish that the bifurcation occurs after day 12. Page numbers are unavailable in the indexed copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

In silico deletion of GATA4 was significantly more deleterious to previously reported highest confidence GATA4 targets than to housekeeping genes.

“In silico deletion of GATA4 was significantly more deleterious to the previously reported highest confidence GATA4 targets33 than to housekeeping genes.”

• cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). sourceSupported

Pretrained Geneformer cell embeddings of cells undergoing iPSC to myoblast differentiation captured the earlier S1 (PAX3+) and later S2B (PAX3+/MYOD+) stages.

“UMAP of pretrained Geneformer cell embeddings of cells undergoing iPSC to myoblast differentiation at the earlier S1 (PAX3+) and later S2B (PAX3+/MYOD+) stages”

• cites [16] Guo, D. et al. iMyoblasts for ex vivo and in vivo investigations of human myogenesis and disease modeling. eLife 11, e70341 (2022). sourcePartly supported

  The Results section on scRNA-seq (page unavailable) identifies an earlier PAX3-expressing S1 cluster and a more differentiated S2B subcluster expressing MYOD1. The paper used Seurat to analyze the cells and visualize clusters with UMAP; it does not report pretrained Geneformer embeddings or establish PAX3 expression specifically in S2B. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Pretrained Geneformer cell embeddings of cells undergoing iPSC reprogramming appropriately captured the temporal trajectory of reprogramming.

“UMAP of pretrained Geneformer cell embeddings of cells undergoing iPSC reprogramming appropriately captured temporal trajectory of reprogramming”

• cites [15] Xing, Q. R. et al. Diversification of reprogramming trajectories revealed by parallel single-cell transcriptome and chromatin accessibility sequencing. Sci. Adv. 6, 463–474 (2020). sourcePartly supported

  The Results section (‘Single-cell profiling of human cell fate reprogramming’; page unavailable) reports that UMAP clustering showed a dynamic transition from parental cells to successfully reprogrammed cells. The source thus supports the temporal reprogramming pattern, but it does not report an analysis of pretrained Geneformer cell embeddings or establish that those embeddings captured it. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

In silico deletion of TBX5 was significantly more deleterious to previously reported TBX5 direct targets than to housekeeping genes or TBX5 indirect targets.

“In silico deletion of TBX5 was significantly more deleterious to previously reported TBX5 direct targets34 than to housekeeping genes or TBX5 indirect targets.”

• cites [34] Kathiriya, I. S. et al. Modeling human TBX5 haploinsufficiency predicts regulatory networks for congenital heart disease. Dev. Cell 56, 292–309 (2021). sourceSupported

The authors assembled Genecorpus-30M, comprised of 29.9 million human single cell transcriptomes from a broad range of tissues from publicly available data.

“We assembled a large-scale pretraining corpus, Genecorpus-30M, comprised of 29.9 million human single cell transcriptomes from a broad range of tissues from publicly available data.”

The authors developed Geneformer, a context-aware, attention-based deep learning model pretrained on large-scale transcriptomic data to enable predictions in settings with limited data.

“Here, we developed a context-aware, attention-based deep learning model, Geneformer, pretrained on large-scale transcriptomic data to enable predictions in settings with limited data.”

• AssumptionRests on an assumption: Mapping gene networks requires large amounts of transcriptomic data to learn the connections between genes, which impedes discoveries in settings with limited data, including rare diseases and diseases affecting clinically inaccessible tissues.

Gene embeddings were dependent on the context of other genes expressed in the cell, highlighting Geneformer's context awareness.

“However, gene embeddings were dependent on the context of other genes expressed in the cell, highlighting Geneformer’s context awareness.”

Geneformer was pretrained on Genecorpus-30M using a self-supervised masked learning objective to gain a fundamental understanding of network dynamics.

“We then pretrained Geneformer on this corpus using a self-supervised masked learning objective to gain a fundamental understanding of network dynamics.”

The pretrained Geneformer accurately predicted dosage-sensitive disease genes and their downstream targets through a context-aware in silico deletion approach.

“The pretrained Geneformer accurately predicted dosage-sensitive disease genes and their downstream targets through a context-aware in silico deletion approach.”

Applied to disease modeling of cardiomyopathy, Geneformer predicted candidate therapeutic targets whose experimental inhibition significantly improved cardiomyocyte contraction in an iPSC-based model of the disease.

“Applied to disease modeling of cardiomyopathy, Geneformer predicted candidate therapeutic targets whose experimental inhibition significantly improved cardiomyocyte contraction in an induced pluripotent stem cell (iPSC)-based model of the disease.”

Fine-tuning Geneformer towards a diverse panel of downstream tasks relevant to chromatin and network dynamics using limited task-specific data demonstrated that Geneformer consistently boosted predictive accuracy.

“Furthermore, fine-tuning Geneformer towards a diverse panel of downstream tasks relevant to chromatin and network dynamics using just a limited set of task-specific training examples demonstrated that Geneformer consistently boosted predictive accuracy.”

Geneformer significantly boosted the ability to predict bivalently marked genes compared to alternative methods (AUC 0.93 and 0.88; bivalent versus unmethylated or H3K4me3-only, respectively).

“Geneformer significantly boosted the ability to predict bivalently marked genes compared to alternative methods (AUC 0.93 and 0.88; bivalent versus unmethylated or H3K4me3-only, respectively) (Fig. 3a–b, Extended Data Fig. 7d–e).”

The fine-tuned Geneformer model correctly predicted the high confidence genes to be dosage-sensitive in the specific context of fetal cerebral cells with 96% concordance with the original study.

“The fine-tuned Geneformer model correctly predicted the high confidence genes to be dosage-sensitive in the specific context of fetal cerebral cells with 96% concordance with the original study.”

In silico deletion of known cardiomyopathy and structural heart disease genes had a significantly larger effect than the control set of known hyperlipidemia genes in fetal cardiomyocytes.

“In silico deletion of known cardiomyopathy and structural heart disease genes had a significantly larger effect than the control set of known hyperlipidemia genes, which are expressed in cardiomyocytes and related to heart disease but whose phenotype affects cell types other than cardiomyocytes (Fig. 2d).”

The final layers were dominated by centrality-driven attention heads that focused on the highest ranked genes that uniquely define each cell state.

“The middle layers were most broad in terms of gene ranks they attended, and the final layers were dominated by centrality-driven attention heads that focused on the highest ranked genes that uniquely define each cell state (Extended Data Fig. 9c–d).”

The middle layers were most broad in terms of gene ranks they attended.

“The middle layers were most broad in terms of gene ranks they attended, and the final layers were dominated by centrality-driven attention heads that focused on the highest ranked genes that uniquely define each cell state (Extended Data Fig. 9c–d).”

Genes whose deletion was predicted to have the most deleterious effect on cardiomyocytes were significantly enriched for human phenotypes including cardiomyopathy and abnormal myocardial morphology.

“Overall, genes whose deletion was predicted to have the most deleterious effect on cardiomyocytes were significantly enriched for human phenotypes including cardiomyopathy and abnormal myocardial morphology (Supplementary Table 3–4).”

More than twice as many general cardiac ECs were needed to gain similar predictive potential as was possible from fine-tuning with the more relevant data from healthy versus dilated aortas.

“More than twice as many general cardiac ECs were needed to gain similar predictive potential as was possible from fine-tuning with the more relevant data from healthy versus dilated aortas, suggesting that the minimum amount of fine-tuning data needed is dependent on both the specific application and relevance of the data to that task.”

Specific attention heads significantly attended central regulatory nodes to a greater degree than peripheral genes within N1-dependent network in ECs.

“Furthermore, specific attention heads significantly attended central regulatory nodes to a greater degree than peripheral genes within N1-dependent network in ECs (Extended Data Fig. 9c).”

Exposure to hundreds of experimental datasets during pretraining appeared to promote robustness to batch-dependent technical artifacts and individual variability that commonly impact single cell analyses in biology.

“Additionally, exposure to hundreds of experimental datasets during pretraining also appeared to promote robustness to batch-dependent technical artifacts and individual variability that commonly impact single cell analyses in biology.”

The fine-tuned Geneformer predicted moderate confidence genes to be dosage-sensitive in fetal cerebral cells with 84% concordance with the original study.

“The fine-tuned Geneformer predicted moderate confidence genes to be dosage-sensitive in fetal cerebral cells with 84% concordance with the original study.”

Geneformer significantly boosted the ability to predict the regulatory range of transcription factors compared to alternative methods, whose predictions were near random.

“Again, Geneformer significantly boosted the ability to predict the regulatory range of transcription factors compared to alternative methods, whose predictions were near random (Fig. 3e, Extended Data Fig. 8d).”

Moderate confidence genes were predicted to be dosage-sensitive at a higher rate in fetal cerebral cells compared to neurons across any adult or developmental timepoint.

“The moderate confidence genes were predicted to be dosage-sensitive at a higher rate in fetal cerebral cells compared to neurons across any adult or developmental timepoint, consistent with these genes’ association with predominantly neurodevelopmental phenotypes where adult neurons may be less relevant.”

High confidence genes were predicted by Geneformer to be dosage-sensitive at similar rates in fetal cerebral (96%) and other cells (95%).

“Interestingly, although the high confidence genes, which may have a stronger effect, were predicted by Geneformer to be dosage-sensitive at similar rates in fetal cerebral (96%) and other cells (95%), the predicted dosage sensitivity of the moderate confidence genes appeared to be more context-specific.”

Geneformer's predictions of bivalently marked genes were generalizable to the remainder of the genome that was excluded from fine-tuning.

“Furthermore, predictions were generalizable to the remainder of the genome that was excluded from fine-tuning (Fig. 3c, Extended Data Fig. 8a–c).”

Fine-tuned Geneformer significantly boosted the ability to predict dosage sensitivity compared to alternative methods (AUC 0.91).

“The fine-tuned Geneformer significantly boosted the ability to predict dosage sensitivity compared to alternative methods (area under the receiver operating characteristic curve (AUC) 0.91) (Fig. 2a, Extended Data Fig. 7a).”

Pretraining with larger and more diverse corpuses consistently improved the predictive power in the downstream dosage sensitivity task despite using the same amount of limited task-specific data for fine-tuning.

“Notably, pretraining with larger and more diverse corpuses consistently improved the predictive power in the downstream task despite using the same amount of limited task-specific data for fine-tuning (Fig. 2b).”

CRISPR-mediated knockout of novel candidate TEAD4 in iPSC-derived cardiac microtissues caused a significant reduction in their ability to generate contractile stress.

“Experimental validation demonstrated that CRISPR-mediated knockout of novel candidate TEAD4 in iPSC-derived cardiac microtissues caused a significant reduction in their ability to generate contractile stress (force per unit area) (Fig. 2e, Extended Data Fig. 7c).”

Attention heads in the earliest layers were consistently the most diverse in terms of gene ranks they attended.

“Interestingly, attention heads in the earliest layers were consistently the most diverse in terms of gene ranks they attended, suggesting the model initially orients to the observed cell state through a joint survey of distinct portions of the input space.”

Pretraining with larger and more diverse corpora consistently improved Geneformer's predictive power.

“Furthermore, we found that pretraining with larger and more diverse corpuses consistently improved Geneformer’s predictive power”

Centrality-driven attention heads consistently attended to a significantly greater degree the highest ranked genes in each cell's unique rank value encoding in aortic ECs, smooth muscle cells, T cells, and macrophage/monocyte/dendritic cells.

“Concordantly, these centrality-driven attention heads consistently attended to a significantly greater degree the highest ranked genes in each cell’s unique rank value encoding in aortic ECs, smooth muscle cells, T cells, and macrophage/monocyte/dendritic cells (which each have different sets of highest ranked genes based on cell type context) (Extended Data Fig. 9d).”

Nearly equivalent predictive potential was retained even when reducing the fine-tuning data to only 5,000 ECs.

“We found that nearly equivalent predictive potential was retained even when reducing the fine-tuning data to only 5,000 ECs (Fig. 4c).”

CRISPR-mediated knockout of both Geneformer-predicted targets GSN and PLN in TTN+/- cells significantly improved the contractile stress of the TTN+/- cardiac microtissues.

“Strikingly, CRISPR-mediated knockout of both Geneformer-predicted targets GSN and PLN in the TTN+/− cells significantly improved the contractile stress of the TTN+/− cardiac microtissues, validating these genes as promising candidate therapeutic targets for this disease (Fig. 6f–g, Extended Data Fig. 10e).”

Geneformer boosted cell type predictions in a variety of tissues, with the gap in performance by accuracy and macro F1 score increasing as the number of cell type classes increased.

“Compared to these alternative methods, Geneformer boosted cell type predictions in a variety of tissues, with the gap in performance by accuracy and macro F1 score increasing as the number of cell type classes increased, indicating that Geneformer was robust in even increasingly complex multiclass prediction applications (Extended Data Fig. 5–6).”

Pretrained Geneformer contextual attention versus gene rank in rank value encoding in indicated aortic cell types, which each have different sets of highest ranked genes based on cell type context.

“Pretrained Geneformer contextual attention versus gene rank in rank value encoding in the indicated aortic cell types, which each have different sets of highest ranked genes based on cell type context”

Geneformer fine-tuned for cell type annotation in indicated human tissues had predictive potential as measured by accuracy and macro F1 score compared to XGBoost (CaSTLe) and deep neural network-based (scDeepSort) methods.

“Predictive potential (as measured by accuracy and macro F1 score) of Geneformer fine-tuned for cell type annotation in the indicated human tissues as compared to XGBoost (CaSTLe) and deep neural network-based (scDeepSort) methods.”

Genes such as transcription factors that may be lowly expressed when they are expressed but highly distinguish cell state move to a higher rank within the encoding.

“Conversely, genes such as transcription factors that may be lowly expressed when they are expressed but highly distinguish cell state will move to a higher rank within the encoding”

• AssumptionThe authors state this as an assumption: A representative range of transcript values would be observed within the cells in which each gene was detected.

The rank value encoding method deprioritizes ubiquitously highly-expressed housekeeping genes by normalizing them to a lower rank.

“Specifically, this method will deprioritize ubiquitously highly-expressed housekeeping genes by normalizing them to a lower rank.”

• AssumptionThe authors state this as an assumption: A representative range of transcript values would be observed within the cells in which each gene was detected.

Inaccurate predictions were predominantly in predicting that cardiomyocyte type 2 was type 1, as expected given the minimal examples of cardiomyocyte type 2 in the Drop-seq data.

“Of note, inaccurate predictions were predominantly in predicting that cardiomyocyte type 2 was type 1, as expected given the minimal examples of cardiomyocyte type 2 in the Drop-seq data.”

Predictive potential on the downstream fine-tuning task was essentially equivalent by AUC, confusion matrices, and F1 score whether task-specific data was included in or excluded from the pretraining corpus.

“Predictive potential on the downstream fine-tuning task was measured by 5-fold cross-validation with these 10,000 cells, demonstrating essentially equivalent results by AUC, confusion matrices, and F1 score.”

QPCR data of CRISPR-mediated knockout of TEAD4 in iPSC-derived cardiomyocytes showed a significant effect (n=3, *p<0.05 by t-test).

“Quantitative PCR (QPCR) data of CRISPR-mediated knockout of TEAD4 in iPSC-derived cardiomyocytes (n=3, *p<0.05 by t-test; center line=median, box limits=upper and lower quartiles, whiskers=1.5x interquartile range, points=experimental replicates).”

Housekeeping genes on average show a trend of a higher normalization factor compared to all genes, resulting in deprioritization in the rank value encoding.

“Housekeeping genes on average show a trend of a higher normalization factor (resulting in deprioritization in the rank value encoding) compared to all genes”

• AssumptionThe authors state this as an assumption: A representative range of transcript values would be observed within the cells in which each gene was detected.

Gene sets significantly associated with hypertrophic or dilated cardiomyopathy states by Geneformer in silico deletion disease modeling significantly overlapped with genes differentially expressed in those respective disease states (differentially expressed vs. non-failing) compared to the overlap of those differentially expressed genes with background genes.

“Gene sets significantly associated with hypertrophic or dilated cardiomyopathy states by Geneformer in silico deletion disease modeling significantly overlapped with genes differentially expressed in those respective disease states (differentially expressed vs. non-failing) compared to the overlap of those differentially expressed genes with background genes”

In silico deletion of certain genes in cardiomyocytes from hypertrophic cardiomyopathy patients significantly shifted embeddings towards the non-failing state and away from the dilated cardiomyopathy state.

“Pathway enrichment for genes whose in silico deletion in cardiomyocytes from hypertrophic cardiomyopathy patients significantly shifted embeddings towards the non-failing state and away from the dilated cardiomyopathy state, suggesting candidate therapeutic targets.”

While original data was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (without fine-tuning) clustered primarily by cell type and phenotype.

“While original data (left) was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (right) (without fine-tuning) clustered primarily by cell type and phenotype.”

The gap in performance of Geneformer compared to alternatives increased as the number of cell type classes increased.

“the gap in performance of Geneformer compared to alternatives increased as the number of cell type classes increased”

27.4 million (27,406,217) cells passed the defined quality filters.

“Ultimately, 27.4 million (27,406,217) cells passed the defined quality filters.”

Specific embedding dimension activations distinguish each lung cell type in the fine-tuned model.

“Specific embedding dimension activations distinguish each lung cell type in the fine-tuned model.”

Compared to in silico differentiation with random genes, in silico differentiation of early-stage myogenic cells by artificially adding MYOD to the front of their rank value encodings significantly shifted the gene embeddings from their earlier state to the embedding of that gene in the later MYOD+ myogenic state.

“Compared to in silico differentiation with random genes, in silico differentiation of the early-stage myogenic cells by artificially adding MYOD to the front of their rank value encodings significantly shifted the gene embeddings from their earlier state to the embedding of that gene in the later MYOD+ myogenic state”

Genecorpus-30M comprises 29.9 million (29,900,531) human single cell transcriptomes from a broad range of tissues from publicly available data.

“We assembled a large-scale pretraining corpus, Genecorpus-30M, comprised of 29.9 million (29,900,531) human single cell transcriptomes from a broad range of tissues from publicly available data”

Cell embeddings generated by pretrained Geneformer without fine-tuning clustered primarily by cell type.

“While original data (left) was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (right) (without fine-tuning) clustered primarily by cell type.”

Pretrained Geneformer attention weights in aortic ECs demonstrated that specific attention heads learned in a completely self-supervised way the relative centrality of the top most central versus most peripheral genes in the N1-dependent gene network.

“Pretrained Geneformer attention weights in aortic ECs demonstrated that specific attention heads learned in a completely self-supervised way the relative centrality of the top most central versus most peripheral genes in the N1-dependent gene network (higher valence=more central)”

Known context-dependent NOTCH genes showed higher variance in their contextual embeddings across variable aortic cell types compared to housekeeping gene GAPDH.

“Known context-dependent NOTCH genes showed higher variance in their contextual embeddings across variable aortic cell types compared to housekeeping gene GAPDH.”

Original data was highly affected by patient batch effect.

“While original data (left) was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (right) (without fine-tuning) clustered primarily by cell type.”

Cell type predictions on DroNc-seq data by the model fine-tuned only on Drop-seq data had out-of-sample accuracy 84%.

“Cell type predictions on the DroNc-seq data by the model fine-tuned only on the Drop-seq data (out of sample accuracy 84%).”

Compared to in silico reprogramming with random genes, in silico reprogramming of fibroblasts by artificially adding OCT4, SOX2, KLF4, and MYC (OSKM) to the front of their rank value encodings significantly shifted the gene embeddings from their initial fibroblast state to the embedding of that gene in the iPSC state.

“Compared to in silico reprogramming with random genes, in silico reprogramming of fibroblasts by artificially adding OCT4, SOX2, KLF4, and MYC (OSKM) to the front of their rank value encodings significantly shifted the gene embeddings from their initial fibroblast state to the embedding of that gene in the iPSC state”

The rank-based approach may be more robust against technical artifacts that may systematically bias absolute transcript counts value while the overall relative ranking of genes within each cell remains more stable.

“Furthermore, this rank-based approach may be more robust against technical artifacts that may systematically bias the absolute transcript counts value while the overall relative ranking of genes within each cell remains more stable.”

Context-awareness is a unique strength of Geneformer's model architecture that allows predictions specific to each cell context.

“Accordingly, context-awareness is a unique strength of Geneformer’s model architecture that allows predictions specific to each cell context.”

Geneformer represents a pretrained deep learning model from which fine-tuning towards a broad range of downstream applications can be pursued to accelerate discovery of key network regulators and candidate therapeutic targets.

“Overall, Geneformer represents a pretrained deep learning model from which fine-tuning towards a broad range of downstream applications can be pursued to accelerate discovery of key network regulators and candidate therapeutic targets.”

In silico treatment analysis using limited data may thus enable therapeutic discovery in innumerable diseases that have been previously impeded by limited data due to being rare or affecting clinically inaccessible tissue.

“In silico treatment analysis using limited data may thus enable therapeutic discovery in innumerable diseases that have been previously impeded by limited data due to being rare or affecting clinically inaccessible tissue.”

These findings provide experimental validation in support of the utility of Geneformer as a tool for discovery of candidate therapeutic targets in human disease.

“These findings provide experimental validation in support of the utility of Geneformer as a tool for discovery of candidate therapeutic targets in human disease.”

Experimental validation of a novel dosage-sensitive gene candidate in fetal cardiomyocytes, TEAD4, supports the utility of Geneformer for driving novel biological insights in human development.

“Experimental validation of a novel dosage-sensitive gene candidate in fetal cardiomyocytes, TEAD4, supports the utility of Geneformer for driving novel biological insights in human development.”

The minimum amount of fine-tuning data needed is dependent on both the specific application and relevance of the data to that task.

“More than twice as many general cardiac ECs were needed to gain similar predictive potential as was possible from fine-tuning with the more relevant data from healthy versus dilated aortas, suggesting that the minimum amount of fine-tuning data needed is dependent on both the specific application and relevance of the data to that task.”

Fine-tuning the pretrained Geneformer model improved predictions even for the higher-order transcription factor property of regulatory range.

“Thus, fine-tuning the pretrained Geneformer model was able to improve predictions even for this higher-order transcription factor property of regulatory range, a particularly challenging characteristic to infer from transcriptional data alone.”

These data suggest that in silico perturbation can be applied to model gene network connections.

“These data suggest that in silico perturbation can be applied to model gene network connections.”

Geneformer was robust in increasingly complex multiclass prediction applications.

“indicating that Geneformer was robust in even increasingly complex multiclass prediction applications.”

Genes whose in silico deletion in cardiomyocytes from hypertrophic cardiomyopathy patients shifted embeddings towards the non-failing state and away from the dilated cardiomyopathy state are candidate therapeutic targets.

“Pathway enrichment for genes whose in silico deletion in cardiomyocytes from hypertrophic cardiomyopathy patients significantly shifted embeddings towards the non-failing state and away from the dilated cardiomyopathy state, suggesting candidate therapeutic targets.”

The imbalance of cardiomyocyte type 1 and 2 between the platforms suggests that these cellular subtypes may be an artifact of variable gene detection between the two platforms.

“The imbalance of cardiomyocyte type 1 and 2 between the platforms also suggests that these cellular subtypes may be an artifact of variable gene detection between the two platforms.”