Transfer learning enables predictions in network biology

In the paper

“Furthermore, genes known to be highly context-dependent, such as NOTCH receptors, showed more variability in their embeddings across variable cell types14 compared to the known housekeeping gene GAPDH (Extended Data Fig. 3).”

What the sources say

• ?Could not verify

  The Results sections describe cell-type-specific gene expression in human aortic tissue, but the passages found do not discuss NOTCH receptors as context-dependent genes. Searches for NOTCH, GAPDH, and embedding variability found no passage addressing the cited point. The embedding comparison is attributed to the citing paper’s Extended Data Fig. 3, not clearly to this source. (LLM, conf=0.72)

  cites [14] Li, Y. et al. Single-cell transcriptome analysis reveals dynamic cell populations and differential gene expression patterns in control and aneurysmal human aortic tissue. Circulation 142, 1374–1388 (2020). sourceExperimental — accuracy not yet measured

In the paper

“Mapping the gene regulatory networks that drive disease progression enables screening for molecules that correct the network by normalizing core regulatory elements, rather than targeting peripheral downstream effectors that may not be disease modifying4,5.”

What the sources say

• Partly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [4] Theodoris, C. V. et al. Human disease modeling reveals integrated transcriptional and epigenetic mechanisms of NOTCH1 haploinsufficiency. Cell 160, 1072–1086 (2015). sourceExperimental — accuracy not yet measured
• Supported

  Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [5] Theodoris, C. V. et al. Network-based screen in iPSC-derived cells reveals therapeutic candidate for heart valve disease. Science 371, eabd0724 (2021). source

In the paper

“Using a publicly available aortic aneurysm dataset14 as a test case, we found that while the original data was impacted by inter-patient variability, Geneformer cell embeddings clustered primarily by cell type and phenotype as opposed to individual patient (Extended Data Fig. 4a).”

What the sources say

• Supported

  The citation identifies the aortic aneurysm dataset used as a test case; the Geneformer clustering result is presented as the citing authors’ own finding. The source paper’s Methods describe single-cell RNA sequencing of aortic tissue from eight patients with ascending thoracic aortic aneurysm and three controls, and state that the data will be made available to other researchers. It therefore supports the dataset identification, without needing to report the Geneformer result. (LLM, conf=0.92)

  cites [14] Li, Y. et al. Single-cell transcriptome analysis reveals dynamic cell populations and differential gene expression patterns in control and aneurysmal human aortic tissue. Circulation 142, 1374–1388 (2020). source

In the paper

“Embeddings of genes in iPSC-derived myogenic cells16 showed similar context awareness with in silico differentiation via MYOD (Extended Data Fig. 2d–e).”

What the sources say

• Supported

  The citation is attached to “iPSC-derived myogenic cells,” identifying the cells used, rather than to the citing authors’ embedding result. The Results section (“Isolation of iMyoblasts by iPSC transgene-free induction and reserve cell selection”) describes generating myogenic iMyoblasts from iPSCs. The source need not report the embedding analysis to support this identification. (LLM, conf=0.91)

  cites [16] Guo, D. et al. iMyoblasts for ex vivo and in vivo investigations of human myogenesis and disease modeling. eLife 11, e70341 (2022). source

In the paper

“When we in silico reprogrammed fibroblasts15 by artificially adding OCT4, SOX2, KLF4, and MYC to the front of their rank value encodings, the remaining genes in the transcriptome significantly shifted their embedding towards the iPSC state (Extended Data Fig. 2b–c).”

What the sources say

• Supported

  The citation is attached to “fibroblasts,” identifying the source of the cells or data used in the citing authors’ in silico analysis. The source paper’s Materials and Methods (Experimental design) states that reprogramming began with BJ fibroblasts and that their cells were profiled by single-cell RNA sequencing. The embedding shift is presented as the citing authors’ own result, not a finding attributed to this source. (LLM, conf=0.90)

  cites [15] Xing, Q. R. et al. Diversification of reprogramming trajectories revealed by parallel single-cell transcriptome and chromatin accessibility sequencing. Sci. Adv. 6, 463–474 (2020). source

In the paper

“In silico deletion of GATA4 had a significantly higher effect on genes known to be most significantly dysregulated by GATA4 variants in a previously reported iPSC disease model of GATA4-related heart defects33 (Extended Data Fig. 9e).”

What the sources say

• Supported

  The Results section, “Attenuated Cardiac Gene Program in Mutant CPCs and CMs” (page unavailable), reports an iPSC-based GATA4 G296S disease model and identifies genes differentially expressed in mutant cells. Citation 33 identifies the previously reported model and its dysregulated genes; the in silico deletion result is the citing paper’s own finding, not a result attributed to this source. (LLM, conf=0.91)

  cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). source

In the paper

“Notably, direct GATA4 targets (as defined by ChIP-seq33) were significantly more impacted by in silico deletion of GATA4 in fetal cardiomyocytes compared to indirect targets (Fig. 5a).”

What the sources say

• Supported

  The citation is attached to “as defined by ChIP-seq,” not to the in silico deletion result, which the citing sentence assigns to its own Figure 5a. In Results (“Genome-Wide Co-Occupancy of GATA4 and TBX5 in Human CMs”; page unavailable), the source reports using ChIP-seq to identify GATA4 binding and validate direct targets. (LLM, conf=0.91)

  cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). source

In the paper

“Indeed, in silico deletion of GATA4 or TBX5 alone had a significantly more deleterious effect on their known co-bound targets33 compared to housekeeping genes.”

What the sources say

• Supported

  The citation is attached to “known co-bound targets.” In Results (Genome-Wide Co-Occupancy of GATA4 and TBX5 in Human CMs), the source identifies gene targets co-bound by GATA4 and TBX5. The in silico deletion comparison is the citing authors’ finding, not the point this citation identifies. (LLM, conf=0.92)

  cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). source

In the paper

“When examining the attention weights in aortic ECs14, we found that 20% of attention heads significantly attended transcription factors more than other genes, indicating that specific attention heads learned, in an entirely self-supervised manner, the relative importance of transcription factors in distinguishing cell states (Fig. 4e).”

What the sources say

• Supported

  The citation is attached to “aortic ECs,” identifying the cells examined, rather than to the citing authors’ 20% attention-head finding. In Results, under “Fibroblasts and Other Nonimmune Cells in Human Ascending Aortic Wall” (page unavailable), the source identifies two clusters of endothelial cells in human aortic tissue. The 20% result is presented as the citing authors’ own analysis. (LLM, conf=0.91)

  cites [14] Li, Y. et al. Single-cell transcriptome analysis reveals dynamic cell populations and differential gene expression patterns in control and aneurysmal human aortic tissue. Circulation 142, 1374–1388 (2020). source

In the paper

“Analogously, in silico deletion of TBX5, another known congenital heart disease gene, in fetal cardiomyocytes23 more significantly impacted its direct targets (as defined by ChIP-seq34) compared to indirect targets and housekeeping genes (Extended Data Fig. 9f).”

What the sources say

• Supported

  The Discussion (page unavailable) describes a human fetal single-cell gene-expression atlas, and the Identification and annotation section (page unavailable) says the profiled organs include the heart. Citation 23 follows “fetal cardiomyocytes” and identifies the source of the fetal-cell data; the citing sentence does not attribute the TBX5 deletion result to this atlas. (LLM, conf=0.82)

  cites [23] Cao, J. et al. A human cell atlas of fetal gene expression. Science 370, 808 (2020). source
• Partly supported

  The Results sections on cardiomyocyte function and transcriptional responses report TBX5 loss-of-function in iPSC-derived cardiomyocytes and an association between TBX5-sensitive genes and TBX5 binding sites. They do not report an in silico deletion in fetal cardiomyocytes. The direct-versus-indirect-target comparison is pointed to the citing paper’s Extended Data Fig. 9f, rather than clearly attributed to this source. (LLM, conf=0.85)

  cites [34] Kathiriya, I. S. et al. Modeling human TBX5 haploinsufficiency predicts regulatory networks for congenital heart disease. Dev. Cell 56, 292–309 (2021). sourceExperimental — accuracy not yet measured

In the paper

“Thus, by fine-tuning Geneformer using solely transcriptional data with only 56 labeled loci in ~15,000 ESCs, the model could predict the results of more recent studies30 that included genome-wide profiling of bivalent domains.”

What the sources say

• Supported

  The citation identifies a study whose results the Geneformer model predicted; it does not attribute Geneformer's training or performance to that study. The Introduction states that the authors mapped H3K4me3 and H3K27me3 across the entire genome in human embryonic stem cells, and the Discussion reports genes bearing both marks. The supplied passages do not provide page numbers. (LLM, conf=0.92)

  cites [30] Pan, G. et al. Whole-genome analysis of histone H3 lysine 4 and lysine 27 methylation in human embryonic stem cells. Cell Stem Cell 1, 299–312 (2007). source

In the paper

“Chen et al. previously systematically integrated thousands of transcription factor binding and histone modification profiles assayed by chromatin immunoprecipitation sequencing (ChIP-seq) with thousands of gene expression profiles to identify two classes of transcription factors with distinct ranges of regulatory influence31.”

What the sources say

• Supported

  Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [31] Chen, C. H. et al. Determinants of transcription factor regulatory range. Nat. Commun. 11, 2472 (2020). source

In the paper

“In silico deletion of genes linked by a prior genome-wide association study24 (GWAS) to cardiac magnetic resonance imaging (MRI) traits relevant to cardiac disease also had a larger effect compared to the control set (Extended Data Fig. 7b).”

What the sources say

• Supported

  The Results section (‘Fifty-seven genetic loci associated with cardiac size and function’) reports a GWAS identifying loci associated with cardiac MRI phenotypes. Citation 24 identifies that prior GWAS; the in silico deletion comparison is presented as the citing paper’s own result (Extended Data Fig. 7b), not as a finding of this source. (LLM, conf=0.91)

  cites [24] Pirruccello, J. P. et al. Analysis of cardiac magnetic resonance imaging in 36,000 individuals yields genetic insights into dilated cardiomyopathy. Nat. Commun. 11, 2254 (2020). source

In the paper

“First, we fine-tuned Geneformer to distinguish cardiomyocytes35 from non-failing hearts (n=9) or hearts affected by hypertrophic (n=11) or dilated (n=9) cardiomyopathy with an overall out-of-sample accuracy of 90% (Fig. 6b, Extended Data Fig. 10a).”

What the sources say

• Supported

  The citation is attached to “cardiomyocytes,” identifying the source of the cardiomyocyte data, not attributing the Geneformer result to Chaffin et al. The paper’s Expression profiles of cardiomyopathies section identifies cardiomyocytes in single-nucleus data from non-failing, hypertrophic and dilated cardiomyopathy hearts. The 90% accuracy is presented as the citing authors’ own result. (LLM, conf=0.92)

  cites [35] Chaffin, M. et al. Single-nucleus profiling of human dilated and hypertrophic cardiomyopathy. Nature 608, 174–180 (2022). source

In the paper

“Interestingly, Geneformer was able to distinguish central versus peripheral factors in the N1-dependent network with fine-tuning on this very minimal data to a better degree than the alternative methods’ predictions trained on the larger dataset of ~30,000 ECs32, demonstrating the strength of pretraining in enabling predictions from increasingly limited data (Fig. 4d, Extended Data Fig. 9b).”

What the sources say

• Partly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  cites [32] Litviňuková, M. et al. Cells of the adult human heart. Nature 588, 455–472 (2020). sourceExperimental — accuracy not yet measured

In the paper

“In silico deletion of TBX5 was significantly more deleterious to previously reported TBX5 direct targets34”

What the sources say

• Partly supported

  The Results section, “Discrete transcriptional responses to reduced TBX5 dosage in cardiomyocytes” (page unavailable), reports that TBX5 occupancy was associated with dosage-vulnerable genes and that 61 of 85 stepwise dose-dependent genes were near TBX5 binding sites. It does not report an in silico TBX5 deletion or a statistically significant comparison showing greater harm to previously reported direct targets. (LLM, conf=0.78)

  cites [34] Kathiriya, I. S. et al. Modeling human TBX5 haploinsufficiency predicts regulatory networks for congenital heart disease. Dev. Cell 56, 292–309 (2021). sourceExperimental — accuracy not yet measured

In the paper

“As the cells pass through each layer, the model successively extrudes them from each other to derive separable embeddings that distinguish the cell types.”

What the sources say

• Partly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  cites [11] Selewa, A. et al. Systematic comparison of high-throughput single-cell and single-nucleus transcriptomes during cardiomyocyte differentiation. Sci. Rep. 10, 1535 (2020). sourceExperimental — accuracy not yet measured

In the paper

“Cell embeddings suggested that cells which do not progress to the iPSC state bifurcate into an alternative fate compared to cells that progress to the iPSC state after the day 12 stage.”

What the sources say

• Partly supported

  The Results section on pseudotemporal trajectories reports branching paths and distinguishes a non-reprogrammed state 8 from a successfully reprogrammed state 9. It also places unsuccessful and successful subgroups as early as day 8, so the copy does not establish that the bifurcation occurs after day 12. Page numbers are unavailable in the indexed copy. (LLM, conf=0.84)

  cites [15] Xing, Q. R. et al. Diversification of reprogramming trajectories revealed by parallel single-cell transcriptome and chromatin accessibility sequencing. Sci. Adv. 6, 463–474 (2020). sourceExperimental — accuracy not yet measured

In the paper

“In silico deletion of GATA4 was significantly more deleterious to the previously reported highest confidence GATA4 targets33 than to housekeeping genes.”

What the sources say

• Supported

  The citation is attached to the previously reported GATA4 targets, not to the citing authors’ in silico deletion result. In Results, under “Genome-Wide Co-Occupancy of GATA4 and TBX5 in Human CMs” (page unavailable), the source identifies putative GATA4 and TBX5 targets using differential expression and nearby co-bound sites. It does not report the deletion-versus-housekeeping comparison, but that comparison is not the part attributed to this source. (LLM, conf=0.86)

  cites [33] Ang, Y. S. et al. Disease model of GATA4 mutation reveals transcription factor cooperativity in human cardiogenesis. Cell 167, 1734–1749 (2016). source

In the paper

“UMAP of pretrained Geneformer cell embeddings of cells undergoing iPSC to myoblast differentiation at the earlier S1 (PAX3+) and later S2B (PAX3+/MYOD+) stages”

What the sources say

• Partly supported

  The Results section on scRNA-seq (page unavailable) identifies an earlier PAX3-expressing S1 cluster and a more differentiated S2B subcluster expressing MYOD1. The paper used Seurat to analyze the cells and visualize clusters with UMAP; it does not report pretrained Geneformer embeddings or establish PAX3 expression specifically in S2B. (LLM, conf=0.88)

  cites [16] Guo, D. et al. iMyoblasts for ex vivo and in vivo investigations of human myogenesis and disease modeling. eLife 11, e70341 (2022). sourceExperimental — accuracy not yet measured

In the paper

“UMAP of pretrained Geneformer cell embeddings of cells undergoing iPSC reprogramming appropriately captured temporal trajectory of reprogramming”

What the sources say

• Partly supported

  The Results section (‘Single-cell profiling of human cell fate reprogramming’; page unavailable) reports that UMAP clustering showed a dynamic transition from parental cells to successfully reprogrammed cells. The source thus supports the temporal reprogramming pattern, but it does not report an analysis of pretrained Geneformer cell embeddings or establish that those embeddings captured it. (LLM, conf=0.87)

  cites [15] Xing, Q. R. et al. Diversification of reprogramming trajectories revealed by parallel single-cell transcriptome and chromatin accessibility sequencing. Sci. Adv. 6, 463–474 (2020). sourceExperimental — accuracy not yet measured

In the paper

“In silico deletion of TBX5 was significantly more deleterious to previously reported TBX5 direct targets34 than to housekeeping genes or TBX5 indirect targets.”

What the sources say

• Supported

  The citation is attached to “previously reported TBX5 direct targets,” not to the citing authors’ in silico deletion comparison. In Results, under “Discrete transcriptional responses to reduced TBX5 dosage in cardiomyocytes” (page unavailable), the source examines published TBX5 occupancy data and identifies dose-sensitive genes near TBX5 binding sites as predicted targets. (LLM, conf=0.87)

  cites [34] Kathiriya, I. S. et al. Modeling human TBX5 haploinsufficiency predicts regulatory networks for congenital heart disease. Dev. Cell 56, 292–309 (2021). source

In the paper

“We assembled a large-scale pretraining corpus, Genecorpus-30M, comprised of 29.9 million human single cell transcriptomes from a broad range of tissues from publicly available data.”

In the paper

“Here, we developed a context-aware, attention-based deep learning model, Geneformer, pretrained on large-scale transcriptomic data to enable predictions in settings with limited data.”

In the paper

“However, gene embeddings were dependent on the context of other genes expressed in the cell, highlighting Geneformer’s context awareness.”

In the paper

“We then pretrained Geneformer on this corpus using a self-supervised masked learning objective to gain a fundamental understanding of network dynamics.”

In the paper

“The pretrained Geneformer accurately predicted dosage-sensitive disease genes and their downstream targets through a context-aware in silico deletion approach.”

In the paper

“Applied to disease modeling of cardiomyopathy, Geneformer predicted candidate therapeutic targets whose experimental inhibition significantly improved cardiomyocyte contraction in an induced pluripotent stem cell (iPSC)-based model of the disease.”

In the paper

“Furthermore, fine-tuning Geneformer towards a diverse panel of downstream tasks relevant to chromatin and network dynamics using just a limited set of task-specific training examples demonstrated that Geneformer consistently boosted predictive accuracy.”

In the paper

“Geneformer significantly boosted the ability to predict bivalently marked genes compared to alternative methods (AUC 0.93 and 0.88; bivalent versus unmethylated or H3K4me3-only, respectively) (Fig. 3a–b, Extended Data Fig. 7d–e).”

In the paper

“The fine-tuned Geneformer model correctly predicted the high confidence genes to be dosage-sensitive in the specific context of fetal cerebral cells with 96% concordance with the original study.”

In the paper

“In silico deletion of known cardiomyopathy and structural heart disease genes had a significantly larger effect than the control set of known hyperlipidemia genes, which are expressed in cardiomyocytes and related to heart disease but whose phenotype affects cell types other than cardiomyocytes (Fig. 2d).”

In the paper

“The middle layers were most broad in terms of gene ranks they attended, and the final layers were dominated by centrality-driven attention heads that focused on the highest ranked genes that uniquely define each cell state (Extended Data Fig. 9c–d).”

In the paper

“The middle layers were most broad in terms of gene ranks they attended, and the final layers were dominated by centrality-driven attention heads that focused on the highest ranked genes that uniquely define each cell state (Extended Data Fig. 9c–d).”

In the paper

“Overall, genes whose deletion was predicted to have the most deleterious effect on cardiomyocytes were significantly enriched for human phenotypes including cardiomyopathy and abnormal myocardial morphology (Supplementary Table 3–4).”

In the paper

“More than twice as many general cardiac ECs were needed to gain similar predictive potential as was possible from fine-tuning with the more relevant data from healthy versus dilated aortas, suggesting that the minimum amount of fine-tuning data needed is dependent on both the specific application and relevance of the data to that task.”

In the paper

“Furthermore, specific attention heads significantly attended central regulatory nodes to a greater degree than peripheral genes within N1-dependent network in ECs (Extended Data Fig. 9c).”

In the paper

“Additionally, exposure to hundreds of experimental datasets during pretraining also appeared to promote robustness to batch-dependent technical artifacts and individual variability that commonly impact single cell analyses in biology.”

In the paper

“The fine-tuned Geneformer predicted moderate confidence genes to be dosage-sensitive in fetal cerebral cells with 84% concordance with the original study.”

In the paper

“Again, Geneformer significantly boosted the ability to predict the regulatory range of transcription factors compared to alternative methods, whose predictions were near random (Fig. 3e, Extended Data Fig. 8d).”

In the paper

“The moderate confidence genes were predicted to be dosage-sensitive at a higher rate in fetal cerebral cells compared to neurons across any adult or developmental timepoint, consistent with these genes’ association with predominantly neurodevelopmental phenotypes where adult neurons may be less relevant.”

In the paper

“Interestingly, although the high confidence genes, which may have a stronger effect, were predicted by Geneformer to be dosage-sensitive at similar rates in fetal cerebral (96%) and other cells (95%), the predicted dosage sensitivity of the moderate confidence genes appeared to be more context-specific.”

In the paper

“Furthermore, predictions were generalizable to the remainder of the genome that was excluded from fine-tuning (Fig. 3c, Extended Data Fig. 8a–c).”

In the paper

“The fine-tuned Geneformer significantly boosted the ability to predict dosage sensitivity compared to alternative methods (area under the receiver operating characteristic curve (AUC) 0.91) (Fig. 2a, Extended Data Fig. 7a).”

In the paper

“Notably, pretraining with larger and more diverse corpuses consistently improved the predictive power in the downstream task despite using the same amount of limited task-specific data for fine-tuning (Fig. 2b).”

In the paper

“Experimental validation demonstrated that CRISPR-mediated knockout of novel candidate TEAD4 in iPSC-derived cardiac microtissues caused a significant reduction in their ability to generate contractile stress (force per unit area) (Fig. 2e, Extended Data Fig. 7c).”

In the paper

“Interestingly, attention heads in the earliest layers were consistently the most diverse in terms of gene ranks they attended, suggesting the model initially orients to the observed cell state through a joint survey of distinct portions of the input space.”

In the paper

“Furthermore, we found that pretraining with larger and more diverse corpuses consistently improved Geneformer’s predictive power”

In the paper

“Concordantly, these centrality-driven attention heads consistently attended to a significantly greater degree the highest ranked genes in each cell’s unique rank value encoding in aortic ECs, smooth muscle cells, T cells, and macrophage/monocyte/dendritic cells (which each have different sets of highest ranked genes based on cell type context) (Extended Data Fig. 9d).”

In the paper

“We found that nearly equivalent predictive potential was retained even when reducing the fine-tuning data to only 5,000 ECs (Fig. 4c).”

In the paper

“Strikingly, CRISPR-mediated knockout of both Geneformer-predicted targets GSN and PLN in the TTN+/− cells significantly improved the contractile stress of the TTN+/− cardiac microtissues, validating these genes as promising candidate therapeutic targets for this disease (Fig. 6f–g, Extended Data Fig. 10e).”

In the paper

“Compared to these alternative methods, Geneformer boosted cell type predictions in a variety of tissues, with the gap in performance by accuracy and macro F1 score increasing as the number of cell type classes increased, indicating that Geneformer was robust in even increasingly complex multiclass prediction applications (Extended Data Fig. 5–6).”

In the paper

“Pretrained Geneformer contextual attention versus gene rank in rank value encoding in the indicated aortic cell types, which each have different sets of highest ranked genes based on cell type context”

In the paper

“Predictive potential (as measured by accuracy and macro F1 score) of Geneformer fine-tuned for cell type annotation in the indicated human tissues as compared to XGBoost (CaSTLe) and deep neural network-based (scDeepSort) methods.”

In the paper

“Conversely, genes such as transcription factors that may be lowly expressed when they are expressed but highly distinguish cell state will move to a higher rank within the encoding”

In the paper

“Specifically, this method will deprioritize ubiquitously highly-expressed housekeeping genes by normalizing them to a lower rank.”

In the paper

“Of note, inaccurate predictions were predominantly in predicting that cardiomyocyte type 2 was type 1, as expected given the minimal examples of cardiomyocyte type 2 in the Drop-seq data.”

In the paper

“Predictive potential on the downstream fine-tuning task was measured by 5-fold cross-validation with these 10,000 cells, demonstrating essentially equivalent results by AUC, confusion matrices, and F1 score.”

In the paper

“Quantitative PCR (QPCR) data of CRISPR-mediated knockout of TEAD4 in iPSC-derived cardiomyocytes (n=3, *p<0.05 by t-test; center line=median, box limits=upper and lower quartiles, whiskers=1.5x interquartile range, points=experimental replicates).”

In the paper

“Housekeeping genes on average show a trend of a higher normalization factor (resulting in deprioritization in the rank value encoding) compared to all genes”

In the paper

“Gene sets significantly associated with hypertrophic or dilated cardiomyopathy states by Geneformer in silico deletion disease modeling significantly overlapped with genes differentially expressed in those respective disease states (differentially expressed vs. non-failing) compared to the overlap of those differentially expressed genes with background genes”

In the paper

“Pathway enrichment for genes whose in silico deletion in cardiomyocytes from hypertrophic cardiomyopathy patients significantly shifted embeddings towards the non-failing state and away from the dilated cardiomyopathy state, suggesting candidate therapeutic targets.”

In the paper

“While original data (left) was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (right) (without fine-tuning) clustered primarily by cell type and phenotype.”

In the paper

“the gap in performance of Geneformer compared to alternatives increased as the number of cell type classes increased”

In the paper

“Ultimately, 27.4 million (27,406,217) cells passed the defined quality filters.”

In the paper

“Specific embedding dimension activations distinguish each lung cell type in the fine-tuned model.”

In the paper

“Compared to in silico differentiation with random genes, in silico differentiation of the early-stage myogenic cells by artificially adding MYOD to the front of their rank value encodings significantly shifted the gene embeddings from their earlier state to the embedding of that gene in the later MYOD+ myogenic state”

In the paper

“We assembled a large-scale pretraining corpus, Genecorpus-30M, comprised of 29.9 million (29,900,531) human single cell transcriptomes from a broad range of tissues from publicly available data”

In the paper

“While original data (left) was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (right) (without fine-tuning) clustered primarily by cell type.”

In the paper

“Pretrained Geneformer attention weights in aortic ECs demonstrated that specific attention heads learned in a completely self-supervised way the relative centrality of the top most central versus most peripheral genes in the N1-dependent gene network (higher valence=more central)”

In the paper

“Known context-dependent NOTCH genes showed higher variance in their contextual embeddings across variable aortic cell types compared to housekeeping gene GAPDH.”

In the paper

“While original data (left) was highly affected by patient batch effect, cell embeddings generated by pretrained Geneformer (right) (without fine-tuning) clustered primarily by cell type.”

In the paper

“Cell type predictions on the DroNc-seq data by the model fine-tuned only on the Drop-seq data (out of sample accuracy 84%).”

In the paper

“Compared to in silico reprogramming with random genes, in silico reprogramming of fibroblasts by artificially adding OCT4, SOX2, KLF4, and MYC (OSKM) to the front of their rank value encodings significantly shifted the gene embeddings from their initial fibroblast state to the embedding of that gene in the iPSC state”

In the paper

“Furthermore, this rank-based approach may be more robust against technical artifacts that may systematically bias the absolute transcript counts value while the overall relative ranking of genes within each cell remains more stable.”

In the paper

“Accordingly, context-awareness is a unique strength of Geneformer’s model architecture that allows predictions specific to each cell context.”

In the paper

“Overall, Geneformer represents a pretrained deep learning model from which fine-tuning towards a broad range of downstream applications can be pursued to accelerate discovery of key network regulators and candidate therapeutic targets.”

In the paper

“In silico treatment analysis using limited data may thus enable therapeutic discovery in innumerable diseases that have been previously impeded by limited data due to being rare or affecting clinically inaccessible tissue.”

In the paper

“These findings provide experimental validation in support of the utility of Geneformer as a tool for discovery of candidate therapeutic targets in human disease.”

In the paper

“Experimental validation of a novel dosage-sensitive gene candidate in fetal cardiomyocytes, TEAD4, supports the utility of Geneformer for driving novel biological insights in human development.”

In the paper

“More than twice as many general cardiac ECs were needed to gain similar predictive potential as was possible from fine-tuning with the more relevant data from healthy versus dilated aortas, suggesting that the minimum amount of fine-tuning data needed is dependent on both the specific application and relevance of the data to that task.”

In the paper

“Thus, fine-tuning the pretrained Geneformer model was able to improve predictions even for this higher-order transcription factor property of regulatory range, a particularly challenging characteristic to infer from transcriptional data alone.”

In the paper

“These data suggest that in silico perturbation can be applied to model gene network connections.”

In the paper

“indicating that Geneformer was robust in even increasingly complex multiclass prediction applications.”

In the paper

“Pathway enrichment for genes whose in silico deletion in cardiomyocytes from hypertrophic cardiomyopathy patients significantly shifted embeddings towards the non-failing state and away from the dilated cardiomyopathy state, suggesting candidate therapeutic targets.”

In the paper

“The imbalance of cardiomyocyte type 1 and 2 between the platforms also suggests that these cellular subtypes may be an artifact of variable gene detection between the two platforms.”