Cited by 327Single-cell and spatial transcriptomicsBioinformatics and Genomic NetworksGenomics and Phylogenetic Studies

Robustness and applicability of transcription factor and pathway analysis tools on single-cell RNA-seq data

Open access
Holland, Christian H., Tanevski, Jovan, Perales-Patón, Javier, Gleixner, Jan, Kumar, Manu P., Mereu, Elisabetta, Joughin, Brian A., Stegle, Oliver +4 more·2020·Genome Biology

Published version of a preprint

Abstract

Background: Many functional analysis tools have been developed to extract functional and mechanistic insight from bulk transcriptome data. With the advent of single-cell RNA sequencing (scRNA-seq), it is in principle possible to do such an analysis for single cells. However, scRNA-seq data has characteristics such as drop-out events and low library sizes. It is thus not clear if functional TF and pathway analysis tools established for bulk sequencing can be applied to scRNA-seq in a meaningful way. Results: To address this question, we perform benchmark studies on simulated and real scRNA-seq data. We include the bulk-RNA tools PROGENy, GO enrichment, and DoRothEA that estimate pathway and transcription factor (TF) activities, respectively, and compare them against the tools SCENIC/AUCell and metaVIPER, designed for scRNA-seq. For the in silico study, we simulate single cells from TF/pathway perturbation bulk RNA-seq experiments. We complement the simulated data with real scRNA-seq data upon CRISPR-mediated knock-out. Our benchmarks on simulated and real data reveal comparable performance to the original bulk data. Additionally, we show that the TF and pathway activities preserve cell type-specific variability by analyzing a mixture sample sequenced with 13 scRNA-seq protocols. We also provide the benchmark data for further use by the community. Conclusions: Our analyses suggest that bulk-based functional analysis tools that use manually curated footprint gene sets can be applied to scRNA-seq data, partially outperforming dedicated single-cell tools. Furthermore, we find that the performance of functional analysis tools is more sensitive to the gene sets than to the statistic used.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13059-020-1949-z

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Enrichment methods developed for bulk transcriptomics can be applied to scRNA-seq.

    “Enrichment methods developed for bulk transcriptomics can be applied to scRNA-seq102”

    From Best practices for single-cell analysis across modalities (2023) · cites this paper as [102]

  • Gene set enrichment analysis is more sensitive to the choice of gene sets than to statistical methods.

    “Gene set enrichment analysis was found to be more sensitive to the choice of gene sets rather than statistical methods102”

    From Best practices for single-cell analysis across modalities (2023) · cites this paper as [102]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [40]Sergushichev A. An algorithm for fast preranked gene set enrichment analysis using cumulative statistic calculation. bioRxiv. 2016:060012 [cited 2018 Jul 17]. Available from: https://www.biorxiv.org/content/early/2016/06/20/060012.abstract.
  2. [51]Holland CH, Tanevski J, Perales-Patón J, Gleixner J, Kumar MP, Mereu E, et al. Robustness and applicability of transcription factor and pathway analysis tools on single-cell RNA-seq data. GitHub. 2020; Available from: https://github.com/saezlab/FootprintMethods_on_scRNAseq. PMC7017576 32051003
  3. [52]Holland CH Saez-Rodriguez J Robustness and applicability of transcription factor and pathway analysis tools on single-cell RNA-seq data 2019 PMC7017576 32051003

Cited by

Sorted by citations

Authors
Holland, C. H., Tanevski, J., Perales-Patón, J., Gleixner, J., Kumar, M. P., Mereu, E., Joughin, B. A., Stegle, O., Lauffenburger, D. A., Heyn, H., Szalai, B., & Saez-Rodriguez, J.
Affiliations
  • Heidelberg University (DE)
  • German Cancer Research Center (DE)
  • Barcelona Institute of Science and Technology (ES)
  • Koch Institute for Integrative Cancer Research At MIT (US)
  • European Bioinformatics Institute (GB)
  • Massachusetts Institute of Technology (US)
  • Universitat Pompeu Fabra (ES)
  • Semmelweis University (HU)
Venue
Genome Biology
Published
2020-02-12
Volume
21
Issue
1
Pages
36-36
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
OpenAlex
W3007199577
Cited by
323
References
51
Citations by year
2026: 37 · 2025: 39 · 2024: 67 · 2023: 60 · 2022: 61 · 2021: 38 · 2020: 18 · 2013: 1
FWCI
14.34
Type
journal article
Language
en
Open access
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (publishedVersion, cc-by-nc)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • Semmelweis University Repository (Semmelweis University) (submittedVersion)
Funders
  • Bundesministerium für Bildung und Forschung (FKZ: 031L0049)
  • National Institutes of Health (U54-CA217377)
  • Ministry of Science, Innovation and Universities (SAF2017-89109-P)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
single-cell RNA sequencing, transcription factor activity, pathway activity, benchmarking, offspring, SCENIC, AUCell, Dorothea, gene ontology enrichment, simulated data
MeSH terms
RNA-Seq, Animals, Humans, Software, Transcription Factors, Benchmarking, Gene Regulatory Networks, Single-Cell Analysis, Transcriptome

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review