scLong: a billion-parameter foundation model for capturing long-range gene context in single-cell transcriptomics
Open accessPublished version of a preprint
Abstract
Single-cell RNA sequencing (scRNA-seq) has revolutionized the study of cellular heterogeneity by providing gene expression data at single-cell resolution, uncovering insights into rare cell populations, cell-cell interactions, and gene regulation. Foundation models pretrained on large-scale scRNA-seq datasets have shown great promise in analyzing such data, but existing approaches are often limited to modeling a small subset of highly expressed genes and lack the integration of external gene-specific knowledge. To address these limitations, we present scLong, a billion-parameter foundation model pretrained on 48 million cells. scLong performs self-attention across the entire set of 28,000 genes in the human genome. This enables the model to capture long-range dependencies between all genes, including lowly expressed ones (containing unexpressed genes with zero expressions), which often play critical roles in cellular processes but are typically excluded by existing foundation models. Additionally, scLong integrates gene knowledge from the Gene Ontology using a graph convolutional network, enriching its contextual understanding of gene functions and relationships. In extensive evaluations, scLong surpasses both state-of-the-art scRNA-seq foundation models and task-specific models across diverse tasks, including predicting transcriptional responses to genetic and chemical perturbations, forecasting cancer drug responses, and inferring gene regulatory networks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530 · see citing works
Inferring Regulatory Networks from Expression Data Using Tree-Based Methods
Huynh-Thu, V. A., Irrthum, A., Wehenkel, L., & Geurts, P. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012776 · Cited by 2,131 · cited 2× · see citing works
“3b) (all P < 0.04, two-sided t -test with sample sizes of 5 and effect sizes greater than 2; see Supplementary Table 12).”
InfluentialDixit, A., Parnas, O., Li, B., Chen, J., Fulco, C. P., Jerby-Arnon, L., Marjanovic, N. D., Dionne, D., Burks, T., Raychowdhury, R., et al. (2016) · Cell · doi:10.1016/j.cell.2016.11.038 · Cited by 1,863 · see citing works
Benchmarking atlas-level data integration in single-cell genomics
Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Mueller, M. F., Strobl, D. C., Zappia, L., Dugas, M., Colomé-Tatché, M., et al. (2022) · Nature Methods · doi:10.1038/s41592-021-01336-8 · Cited by 1,438 · see citing works
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · cited 2× · see citing works
“One key limitation is that, to save computational cost, these models typically perform self-attention on a small subset of genes (e.g., 2048 in Geneformer 8 and scFoundation 10 , 2000 in scGPT 9 , and 1536 in UCE 16 ), often selected based on high expression levels 8 .”
A single-cell transcriptomic atlas characterizes ageing tissues in the mouse
The Tabula Muris Consortium, Almanzar, N., Antony, J., Baghel, A. S., Bakerman, I., Bansal, I., Barres, B. A., Beachy, P. A., Berdnik, D., Bilen, B., et al. (2020) · Nature · doi:10.1038/s41586-020-2496-1 · Cited by 1,247 · see citing works
Single-cell messenger RNA sequencing reveals rare intestinal cell types
Grün, D., Lyubimova, A., Kester, L., Wiebrands, K., Basak, O., Sasaki, N., Clevers, H., & van Oudenaarden, A. (2015) · Nature · doi:10.1038/nature14966 · Cited by 1,234 · see citing works
Transfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · see citing works
Construction of a human cell landscape at single-cell level
Han, X., Zhou, Z., Fei, L., Sun, H., Wang, R., Chen, Y., Chen, H., Wang, J., Tang, H., Ge, W., et al. (2020) · Nature · doi:10.1038/s41586-020-2157-4 · Cited by 1,087 · see citing works
Self-citationCombinatorial drug therapy for cancer in the post-genomic era
Al-Lazikani, B., Banerji, U., & Workman, P. (2012) · Nature Biotechnology · doi:10.1038/nbt.2284 · Cited by 1,070 · see citing works
Best practices for single-cell analysis across modalities
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00586-w · Cited by 1,051 · see citing works
Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data
Pratapa, A., Jalihal, A. P., Law, J. N., Bharadwaj, A., & Murali, T. M. (2020) · Nature Methods · doi:10.1038/s41592-019-0690-6 · Cited by 792 · see citing works
ChIP‐Atlas: a data‐mining suite powered by full integration of public ChIP‐seq data
Oki, S., Ohta, T., Shioi, G., Hatanaka, H., Ogasawara, O., Okuda, Y., Kawaji, H., Nakaki, R., Sese, J., & Meno, C. (2018) · The EMBO Reports · doi:10.15252/embr.201846255 · Cited by 763 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
Large Scale Foundation Model on Single-cell Transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Song, L., & Zhang, X. (2023) · doi:10.1101/2023.05.29.542705 · Cited by 639 · see citing works
Large-scale foundation model on single-cell transcriptomics
Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Zhang, X., & Song, L. (2024) · Nature Methods · doi:10.1038/s41592-024-02305-7 · Cited by 628 · cited 4× · see citing works
“Current models 8 – 10 rely predominantly on patterns derived solely from gene expression data, which restricts their ability to capture context related togenefunctionsandregulatoryinteractions.”
InfluentialPredicting transcriptional outcomes of novel multigene perturbations with GEARS
Roohani, Y., Huang, K., & Leskovec, J. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01905-6 · Cited by 355 · see citing works
Ahlmann-Eltze, C., Huber, W., & Anders, S. (2025) · Nature Methods · doi:10.1038/s41592-025-02772-6 · Cited by 238 · see citing works
Yang, X., Liu, G., Feng, G., Bu, D., Wang, P., Jiang, J., Chen, S., Yang, Q., Zhang, Y., Man, Z., et al. (2023) · doi:10.1101/2023.09.26.559542 · Cited by 148 · see citing works
Zero-shot evaluation reveals limitations of single-cell foundation models
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2025) · Genome Biology · doi:10.1186/s13059-025-03574-x · Cited by 109 · cited 5× · see citing works
“…scLong with baselines from 53 , including: (1) Raw , which directly uses the raw expression values without performing batch integration; (2) HVG 53 , which selects the top 2000 highly variable genes (HVGs) and retains only their expression valuesforeachcell;and(3)atask-speci fi cmethodscVI 54…”
Influential
Flagged first, then by citations
- Authors
- Bai, D., Mo, S., Zhang, R., Luo, Y., Gao, J., Yang, J. P., Wu, Q., Rahmani, H., Amariuta, T., Grotjahn, D., Zhong, S., Lewis, N., Wang, W., Ideker, T., Xie, P., & Xing, E.
- Affiliations
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- University of California San Diego (US)
- Carnegie Mellon University (US)
- Scripps Research Institute (US)
- Corresponding author
- Xie, Pengtao
- Xing, Eric
- ORCID
- Venue
- Nature Communications
- Published
- 2026-02-05
- Volume
- 17
- Issue
- 1
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC12982784
- OpenAlex
- W7128035223
- Cited by
- 15
- References
- 22
- Citations by year
- 2026: 10 · 2025: 5
- FWCI
- 20.67
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nature Communications (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Keywords
- single-cell RNA sequencing, foundation models, self-attention, lowly expressed genes, graph convolutional networks, cancer drug response prediction, gene regulatory network inference
- MeSH terms
- RNA-Seq, Single-Cell Gene Expression Analysis, Humans, Models, Genetic, Software, Genome, Human, Sequence Analysis, RNA, Computational Biology, Gene Expression Profiling, Gene Regulatory Networks, Single-Cell Analysis, Transcriptome, Gene Ontology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
