The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data
Abstract
A bstract: The advances in high-throughput sequencing technology have led to significant progress in measuring gene expressions in single-cell level. The amount of publicly available single-cell RNA-seq (scRNA-seq) data is already surpassing 50M records for human with each record measuring 20,000 genes. This highlights the need for unsupervised representation learning to fully ingest these data, yet classical transformer architectures are prohibitive to train on such data in terms of both computation and memory. To address this challenge, we propose a novel asymmetric encoder-decoder transformer for scRNA-seq data, called xTrimoGene, which leverages the sparse characteristic of the data to scale up the pre-training. This scalable design of xTrimoGene reduces FLOPs by one to two orders of magnitude compared to classical transformers while maintaining high accuracy, enabling us to train the largest transformer models over the largest scRNA-seq dataset today. Our experiments also show that the performance of xTrimoGene improves as we increase the model sizes, and it also leads to SOTA performance over various downstream tasks, such as cell classification, perturb-seq effect prediction, and drug combination prediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We constructed a comprehensive single-cell gene expression data set by collecting data from all publicly available single-cell resources, including GEO 17 , Single Cell Portal, HCA 4 , EMBL-EBI 18 , etc.”
From Large-scale foundation model on single-cell transcriptomics (2024) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“The Deep Learning community has shown significant interest in the scalability of proposed models [18, 2].”
BackgroundDeep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530
Dixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863
Exploring genetic interaction manifolds constructed from rich single-cell phenotypes
Norman, T. M., Horlbeck, M. A., Replogle, J. M., Ge, A. Y., Xu, A., Jost, M., Gilbert, L. A., & Weissman, J. S. (2019) · Science · doi:10.1126/science.aax4438 · Cited by 392
ACTINN: automated identification of cell types in single cell RNA sequencing
Ma, F., & Pellegrini, M. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz592 · Cited by 252
Chen, J., Hu, Z., Sun, S., Tan, Q., Wang, Y., Yu, Q., Zong, L., Hong, L., Xiao, J., Shen, T., et al. (2022) · doi:10.1101/2022.08.06.503062 · Cited by 247
Research works printed but not linked: 32
- Zeming Lin , Halil Akin , Roshan Rao , Brian Hie , Zhongkai Zhu , Wenting Lu , Nikita Smetanin , Robert Verkuil , Ori Kabeli , Yaniv Shmueli , Allan dos Santos Costa , Maryam Fazel-Zarandi , Tom Sercu , Salvatore Candido , and Alexander Rives . Evolutionary-scale prediction of atomic level protein structure with a language model. bioRxiv, 2022.
- 2022, ProteinBERT: a universal deep-learning model of protein sequence and function, Bioinformatics, 38, 2102, 10.1093/bioinformatics/btac020 · DOI · Paper page
- 2022, deepSimDEF: deep neural embeddings of gene products and gene ontology terms for functional analysis of genes, Bioinformatics, 38, 3051, 10.1093/bioinformatics/btac304 · DOI
- 2022, Single-cell rna sequencing technologies and applications: A brief overview, Clinical and Translational Medicine, 12, e694, 10.1002/ctm2.694 · DOI
- Geng Chen , Baitang Ning , and Tieliu Shi . Single-cell rna-seq technologies and related computational data analysis. Frontiers in genetics, page 317, 2019. · DOI
- 2022, heca: The cell-centric assembly of a cell atlas, Iscience, 25, 104318, 10.1016/j.isci.2022.104318 · DOI · Paper page
- 2022, Disco: a database of deeply integrated human single-cell omics data, Nucleic acids research, 50, D596, 10.1093/nar/gkab1020 · DOI · Paper page
- 2022, Deep learning tackles single-cell analysis—a survey of deep learning for scrna-seq analysis, Briefings in bioinformatics, 23, bbab531, 10.1093/bib/bbab531 · DOI
- 2020, scVAE: variational auto-encoders for single-cell gene expression data, Bioinformatics, 36, 4415, 10.1093/bioinformatics/btaa293 · DOI
- 2020, Realistic in silico generation and augmentation of single-cell rna-seq data using generative adversarial networks, Nature Communications, 11, 786
- 2019, Data denoising with transfer learning in single-cell transcriptomics, Nature methods, 16, 875, 10.1038/s41592-019-0537-1 · DOI
- 2020, Pre-trained models for natural language processing: A survey, Science China Technological Sciences, 63, 1872, 10.1007/s11431-020-1647-3 · DOI · Paper page
- 10.1016/j.aiopen.2021.08.002 · DOI · Paper page
- 2019, Proceedings of the 36th International Conference on Machine Learning, 97, 2712
- 2022, scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data, Nature Machine Intelligence, 4, 852, 10.1038/s42256-022-00534-z · DOI · Paper page
- 10.1186/s13059-017-1381-1 · DOI · Paper page
- Samyam Rajbhandari , Jeff Rasley , Olatunji Ruwase , and Yuxiong He . Zero: Memory optimizations toward training trillion parameter models. ArXiv, May 2020. · DOI
- Tianqi Chen , Bing Xu , Chiyuan Zhang , and Carlos Guestrin . Training deep nets with sublinear memory cost. CoRR, abs/1604.06174, 2016. · arXiv
- Yi Tay , Mostafa Dehghani , Samira Abnar , Hyung Won Chung , William Fedus , Jinfeng Rao , Sharan Narang , Vinh Q. Tran , Dani Yogatama , and Donald Metzler . Scaling laws vs model architectures: How does inductive bias influence scaling?, 2022. · DOI · Paper page
- Oscar Franzén , Li-Ming Gan , and Johan L M Björkegren . PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data. Database, 2019, 04 2019. ISSN 1758-0463. baz046. · DOI · Paper page
- 10.1186/1471-2105-14-1 · DOI
- 10.1093/nar/gkw377 · DOI · Paper page
- 10.1529/biophysj.103.029884 · DOI
- 2020, Pd-1/pd-l1 pathway: current researches in cancer, American journal of cancer research, 10, 727
- 10.1016/j.yjmcc.2014.02.017 · DOI
- 2018, Fibroblast growth factor 21 for non-alcoholic steatohepatitis, The Lancet, 392, 2658, 10.1016/S0140-6736(18)32165-2 · DOI
- 2017, Massively parallel digital transcriptional profiling of single cells, Nature communications, 8, 14049, 10.1038/ncomms14049 · DOI · Paper page
- Yusuf Roohani , Kexin Huang , and Jure Leskovec . Gears: Predicting transcriptional outcomes of novel multi-gene perturbations. bioRxiv, 2022. · DOI
- Mohammad Lotfollahi , Anna Klimovskaia Susmelj , Carlo De Donno , Yuge Ji , Ignacio L. Ibarra , F. Alexander Wolf , Nafissa Yakubova , Fabian J. Theis , and David Lopez-Paz . Compositional perturbation autoencoder for single-cell response modeling. bioRxiv, 2021. · DOI
- 10.18632/oncotarget.16723 · DOI · Paper page
- 2017, DeepSynergy: predicting anti-cancer drug synergy with Deep Learning, Bioinformatics, 34, 1538
- 2022, Deepdds: deep graph neural network with attention mechanism to predict synergistic drug combinations, Briefings in Bioinformatics, 23, bbab390, 10.1093/bib/bbab390 · DOI
Cited by
Sorted by citations
Large-scale foundation model on single-cell transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Zhang, X., & Song, L. (2024) · Nature Methods · doi:10.1038/s41592-024-02305-7 · Cited by 628
Self-citationRood, J. E., Hupalowska, A., & Regev, A. (2024) · Cell · doi:10.1016/j.cell.2024.07.035 · Cited by 139
Large Scale Foundation Model on Single-cell Transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Song, L., & Zhang, X. (2023) · doi:10.1101/2023.05.29.542705 · Cited by 639
Self-citation
Metadata
- Authors
- Gong, J., Hao, M., Zeng, X., Liu, C., Ma, J., Cheng, X., Wang, T., Zhang, X., & Song, L.
- Publisher
- openRxiv
- Published
- 2023-03-25
- Cited by
- 17
- References
- 8
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
