PreprintCited by 64Single-cell and spatial transcriptomicsGene expression and cancer classificationGene Regulatory Network Analysis

Evaluating the Utilities of Foundation Models in Single-cell Data Analysis

Open access
Liu, Tianyu, Li, Kexing, Wang, Yuge, Li, Hongyu, Zhao, Hongyu·2023·Advanced Science

Preprint — published as 10.1002/advs.202514490

Abstract

Foundation Models (FMs) have made significant strides in both industrial and scientific domains. In this paper, we evaluate the performance of FMs for single-cell sequencing data analysis through comprehensive experiments across eight downstream tasks pertinent to single-cell data. Overall, the top FMs include scGPT, Geneformer, and CellPLM by considering model performances and user accessibility among ten single-cell FMs. However, by comparing these FMs with task-specific methods, we found that single-cell FMs may not consistently excel than task-specific methods in all tasks, which challenges the necessity of developing foundation models for single-cell analysis. In addition, we evaluated the effects of hyper-parameters, initial settings, and stability for training single-cell FMs based on a proposed scEval framework, and provide guidelines for pre-training and fine-tuning, to enhance the performances of single-cell FMs. Our work summarizes the current state of single-cell FMs, points to their constraints and avenues for future development, and offers a freely available evaluation pipeline to benchmark new models and improve method development.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.09.08.555192

Cites

154 of 154 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our results are in line with previously published benchmarks that found negligible benefits of foundation models compared to simpler approaches for other tasks.

    “Our results are in line with previously published benchmarks that assessed the performance of foundation models for other tasks and found negligible benefits compared to simpler approaches17–19.”

    From Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines (2025) · cites this paper as [19]

    The Overview of our evaluations and Discussion (pages unavailable) show that this paper benchmarked single-cell foundation models against task-specific methods and found that scGPT did not outperform them on batch-effect correction. But the paper does not find negligible benefits across tasks: the Discussion also reports advantages for gene-function prediction and spatial transcriptomic imputation. The claim overgeneralizes its mixed results. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Wang, T., Roberts, A., Hesslow, D., Le Scao, T., Chung, H.W., Beltagy, I., Lau- nay, J., Raffel, C.: What language model architecture and pretraining objective works best for zero-shot generalization? In: International Conference on Machine Learning, pp. 22964-22984 (2022). PMLR
  2. Wang, Y., Si, S., Li, D., Lukasik, M., Yu, F., Hsieh, C.-J., Dhillon, I.S., Kumar, S.: Preserving in-context learning ability in large language model fine-tuning. arXiv preprint arXiv:2211.00635 (2022) · arXiv
  3. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., Fedus, W.: Emergent abilities of large language models. Transactions on Machine Learning Research (2022). Survey Certification
  4. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  5. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Ghahra- mani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27. Curran Asso- ciates, Inc., ??? (2014). https://proceedings.neurips.cc/paper files/paper/2014/ file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

Cited by

Sorted by citations

Authors
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H.
Affiliations
Yale University (US)
Corresponding author
Zhao, Hongyu
Venue
Advanced Science
Publisher
openRxiv
Published
2023-09-08
Volume
13
Issue
27
Pages
e14490-e14490
ISSN
2198-3844 (ISSN-L)
OpenAlex
W7140132101
Cited by
62
References
141
Citations by year
2026: 18 · 2025: 32 · 2024: 10 · 2023: 1
FWCI
37.20
Type
journal article
Language
en
Open access
Locations
  • Advanced Science (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed Central (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (1u01hg013840-01)
  • National Cancer Institute (P50CA196530)
  • National Institute of General Medical Sciences (R01GM134005)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
Keywords
foundation models, single-cell RNA sequencing, scGPT, Geneformer, model benchmarking, pre-training and fine-tuning, hyperparameter effects, model stability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review