CoChat
Sign Up
PreprintCited by 17

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data

Gong, Jing, Hao, Minsheng, Zeng, Xin, Liu, Chiming, Ma, Jianzhu, Cheng, Xingyi, Wang, Taifeng, Zhang, Xuegong, Song, Le·2023
DOI 10.1101/2023.03.24.534055
Open this paper in CoChat — free account

Abstract

A bstract: The advances in high-throughput sequencing technology have led to significant progress in measuring gene expressions in single-cell level. The amount of publicly available single-cell RNA-seq (scRNA-seq) data is already surpassing 50M records for human with each record measuring 20,000 genes. This highlights the need for unsupervised representation learning to fully ingest these data, yet classical transformer architectures are prohibitive to train on such data in terms of both computation and memory. To address this challenge, we propose a novel asymmetric encoder-decoder transformer for scRNA-seq data, called xTrimoGene, which leverages the sparse characteristic of the data to scale up the pre-training. This scalable design of xTrimoGene reduces FLOPs by one to two orders of magnitude compared to classical transformers while maintaining high accuracy, enabling us to train the largest transformer models over the largest scRNA-seq dataset today. Our experiments also show that the performance of xTrimoGene improves as we increase the model sizes, and it also leads to SOTA performance over various downstream tasks, such as cell classification, perturb-seq effect prediction, and drug combination prediction.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.03.24.534055

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 32

  1. Zeming Lin , Halil Akin , Roshan Rao , Brian Hie , Zhongkai Zhu , Wenting Lu , Nikita Smetanin , Robert Verkuil , Ori Kabeli , Yaniv Shmueli , Allan dos Santos Costa , Maryam Fazel-Zarandi , Tom Sercu , Salvatore Candido , and Alexander Rives . Evolutionary-scale prediction of atomic level protein structure with a language model. bioRxiv, 2022.
  2. 2022, ProteinBERT: a universal deep-learning model of protein sequence and function, Bioinformatics, 38, 2102, 10.1093/bioinformatics/btac020 · DOI · Paper page
  3. 2022, deepSimDEF: deep neural embeddings of gene products and gene ontology terms for functional analysis of genes, Bioinformatics, 38, 3051, 10.1093/bioinformatics/btac304 · DOI
  4. 2022, Single-cell rna sequencing technologies and applications: A brief overview, Clinical and Translational Medicine, 12, e694, 10.1002/ctm2.694 · DOI
  5. Geng Chen , Baitang Ning , and Tieliu Shi . Single-cell rna-seq technologies and related computational data analysis. Frontiers in genetics, page 317, 2019. · DOI
  6. 2022, heca: The cell-centric assembly of a cell atlas, Iscience, 25, 104318, 10.1016/j.isci.2022.104318 · DOI · Paper page
  7. 2022, Disco: a database of deeply integrated human single-cell omics data, Nucleic acids research, 50, D596, 10.1093/nar/gkab1020 · DOI · Paper page
  8. 2022, Deep learning tackles single-cell analysis—a survey of deep learning for scrna-seq analysis, Briefings in bioinformatics, 23, bbab531, 10.1093/bib/bbab531 · DOI
  9. 2020, scVAE: variational auto-encoders for single-cell gene expression data, Bioinformatics, 36, 4415, 10.1093/bioinformatics/btaa293 · DOI
  10. 2020, Realistic in silico generation and augmentation of single-cell rna-seq data using generative adversarial networks, Nature Communications, 11, 786
  11. 2019, Data denoising with transfer learning in single-cell transcriptomics, Nature methods, 16, 875, 10.1038/s41592-019-0537-1 · DOI
  12. 2020, Pre-trained models for natural language processing: A survey, Science China Technological Sciences, 63, 1872, 10.1007/s11431-020-1647-3 · DOI · Paper page
  13. 10.1016/j.aiopen.2021.08.002 · DOI · Paper page
  14. 2019, Proceedings of the 36th International Conference on Machine Learning, 97, 2712
  15. 2022, scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data, Nature Machine Intelligence, 4, 852, 10.1038/s42256-022-00534-z · DOI · Paper page
  16. 10.1186/s13059-017-1381-1 · DOI · Paper page
  17. Samyam Rajbhandari , Jeff Rasley , Olatunji Ruwase , and Yuxiong He . Zero: Memory optimizations toward training trillion parameter models. ArXiv, May 2020. · DOI
  18. Tianqi Chen , Bing Xu , Chiyuan Zhang , and Carlos Guestrin . Training deep nets with sublinear memory cost. CoRR, abs/1604.06174, 2016. · arXiv
  19. Yi Tay , Mostafa Dehghani , Samira Abnar , Hyung Won Chung , William Fedus , Jinfeng Rao , Sharan Narang , Vinh Q. Tran , Dani Yogatama , and Donald Metzler . Scaling laws vs model architectures: How does inductive bias influence scaling?, 2022. · DOI · Paper page
  20. Oscar Franzén , Li-Ming Gan , and Johan L M Björkegren . PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data. Database, 2019, 04 2019. ISSN 1758-0463. baz046. · DOI · Paper page
  21. 10.1186/1471-2105-14-1 · DOI
  22. 10.1093/nar/gkw377 · DOI · Paper page
  23. 10.1529/biophysj.103.029884 · DOI
  24. 2020, Pd-1/pd-l1 pathway: current researches in cancer, American journal of cancer research, 10, 727
  25. 10.1016/j.yjmcc.2014.02.017 · DOI
  26. 2018, Fibroblast growth factor 21 for non-alcoholic steatohepatitis, The Lancet, 392, 2658, 10.1016/S0140-6736(18)32165-2 · DOI
  27. 2017, Massively parallel digital transcriptional profiling of single cells, Nature communications, 8, 14049, 10.1038/ncomms14049 · DOI · Paper page
  28. Yusuf Roohani , Kexin Huang , and Jure Leskovec . Gears: Predicting transcriptional outcomes of novel multi-gene perturbations. bioRxiv, 2022. · DOI
  29. Mohammad Lotfollahi , Anna Klimovskaia Susmelj , Carlo De Donno , Yuge Ji , Ignacio L. Ibarra , F. Alexander Wolf , Nafissa Yakubova , Fabian J. Theis , and David Lopez-Paz . Compositional perturbation autoencoder for single-cell response modeling. bioRxiv, 2021. · DOI
  30. 10.18632/oncotarget.16723 · DOI · Paper page
  31. 2017, DeepSynergy: predicting anti-cancer drug synergy with Deep Learning, Bioinformatics, 34, 1538
  32. 2022, Deepdds: deep graph neural network with attention mechanism to predict synergistic drug combinations, Briefings in Bioinformatics, 23, bbab390, 10.1093/bib/bbab390 · DOI

Cited by

Sorted by citations

See all 17

Metadata

Authors
Gong, J., Hao, M., Zeng, X., Liu, C., Ma, J., Cheng, X., Wang, T., Zhang, X., & Song, L.
ORCID
Publisher
openRxiv
Published
2023-03-25
DOI
10.1101/2023.03.24.534055
Cited by
17
References
8
Type
Preprint

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data. https://app.cochat.ai/papers/10.1101/2023.03.24.534055
The paper
Gong, J., Hao, M., Zeng, X., Liu, C., Ma, J., Cheng, X., Wang, T., Zhang, X., & Song, L. (2023). xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data. https://doi.org/10.1101/2023.03.24.534055