Single-Cell RNA-Sequencing: Assessment of Differential Expression Analysis Methods
Open accessAbstract
The sequencing of the transcriptomes of single-cells, or single-cell RNA-sequencing, has now become the dominant technology for the identification of novel cell types and for the study of stochastic gene expression. In recent years, various tools for analyzing single-cell RNA-sequencing data have been proposed, many of them with the purpose of performing differentially expression analysis. In this work, we compare four different tools for single-cell RNA-sequencing differential expression, together with two popular methods originally developed for the analysis of bulk RNA-sequencing data, but largely applied to single-cell data. We discuss results obtained on two real and one synthetic dataset, along with considerations about the perspectives of single-cell differential expression analysis. In particular, we explore the methods performance in four different scenarios, mimicking different unimodal or bimodal distributions of the data, as characteristic of single-cell transcriptomics. We observed marked differences between the selected methods in terms of precision and recall, the number of detected differentially expressed genes and the overall performance. Globally, the results obtained in our study suggest that is difficult to identify a best performing tool and that efforts are needed to improve the methodologies for single-cell RNA-sequencing data analysis and gain better accuracy of results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Tools built specifically to handle single-cell data do not appear to perform better than tools created for bulk data analysis.
“The result is reduced reproducibility with real data, leading to the conclusion that tools built specifically to handle single-cell data do not appear to perform better than tools created for bulk data analysis22–24.”
From A practical solution to pseudoreplication bias in single-cell studies (2021) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106 · see citing works
Robinson, M. D., McCarthy, D. J., & Smyth, G. K. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btp616 · Cited by 41,364 · see citing works
Differential expression analysis for sequence count data
Anders, S., & Huber, W. (2010) · Genome Biology · doi:10.1186/gb-2010-11-10-r106 · Cited by 15,581 · see citing works
Use of Ranks in One-Criterion Variance Analysis
Kruskal, W. H., & Wallis, W. A. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483441 · Cited by 13,912 · see citing works
Smyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829 · see citing works
Trapnell, C., Cacchiarelli, D., Grimsby, J., Pokharel, P., Li, S., Morse, M., Lennon, N. J., Livak, K. J., Mikkelsen, T. S., & Rinn, J. L. (2014) · Nature Biotechnology · doi:10.1038/nbt.2859 · Cited by 7,314 · see citing works
Finak, G., McDavid, A., Yajima, M., Deng, J., Gersuk, V., Shalek, A. K., Slichter, C. K., Miller, H. W., McElrath, M. J., Prlic, M., et al. (2015) · Genome Biology · doi:10.1186/s13059-015-0844-5 · Cited by 3,159 · see citing works
Stochastic mRNA Synthesis in Mammalian Cells
Raj, A., Peskin, C. S., Tranchina, D., Vargas, D. Y., & Tyagi, S. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040309 · Cited by 1,737 · see citing works
Adult mouse cortical cell taxonomy revealed by single cell transcriptomics
Tasic, B., Menon, V., Nguyen, T. N., Kim, T. K., Jarsky, T., Yao, Z., Levi, B., Gray, L. T., Sorensen, S. A., Dolbeare, T., et al. (2016) · Nature Neuroscience · doi:10.1038/nn.4216 · Cited by 1,714 · see citing works
The Technology and Biology of Single-Cell RNA Sequencing
Kolodziejczyk, A. A., Kim, J. K., Svensson, V., Marioni, J. C., & Teichmann, S. A. (2015) · Molecular Cell · doi:10.1016/j.molcel.2015.04.005 · Cited by 1,487 · see citing works
Bayesian approach to single-cell differential expression analysis
Kharchenko, P. V., Silberstein, L., & Scadden, D. T. (2014) · Nature Methods · doi:10.1038/nmeth.2967 · Cited by 1,348 · see citing works
Single-cell transcriptomics reveals bimodality in expression and splicing in immune cells
Shalek, A. K., Satija, R., Adiconis, X., Gertner, R. S., Gaublomme, J. T., Raychowdhury, R., Schwartz, S., Yosef, N., Malboeuf, C., Lu, D., et al. (2013) · Nature · doi:10.1038/nature12172 · Cited by 1,190 · see citing works
Computational and analytical challenges in single-cell transcriptomics
Stegle, O., Teichmann, S. A., & Marioni, J. C. (2015) · Nature Reviews Genetics · doi:10.1038/nrg3833 · Cited by 1,170 · see citing works
Small-sample estimation of negative binomial dispersion, with applications to SAGE data
Robinson, M. D., & Smyth, G. K. (2007) · Biostatistics · doi:10.1093/biostatistics/kxm030 · Cited by 1,073 · see citing works
Characterization of the single-cell transcriptional landscape by highly multiplex RNA-seq
Islam, S., Kjällquist, U., Moliner, A., Zajac, P., Fan, J.-B., Lönnerberg, P., & Linnarsson, S. (2011) · Genome Research · doi:10.1101/gr.110882.110 · Cited by 949 · see citing works
Markovian Modeling of Gene-Product Synthesis
Peccoud, J., & Ycart, B. (1995) · Theoretical Population Biology · doi:10.1006/tpbi.1995.1027 · Cited by 739 · see citing works
Validation of noise models for single-cell transcriptomics
Grün, D., Kester, L., & van Oudenaarden, A. (2014) · Nature Methods · doi:10.1038/nmeth.2930 · Cited by 721 · see citing works
Transcriptional Pulsing of a Developmental Gene
Chubb, J. R., Trcek, T., Shenoy, S. M., & Singer, R. H. (2006) · Current Biology · doi:10.1016/j.cub.2006.03.092 · Cited by 714 · see citing works
Individual Comparisons of Grouped Data by Ranking Methods
Wilcoxon, F. (1946) · Journal of Economic Entomology · doi:10.1093/jee/39.2.269 · Cited by 482 · see citing works
Design and computational analysis of single-cell RNA-sequencing experiments
Bacher, R. L., & Kendziorski, C. M. (2016) · Genome biology · doi:10.1186/s13059-016-0927-y · Cited by 437 · see citing works
Single-Cell Transcriptomics of the Human Endocrine Pancreas
Wang, Y. J., Schug, J., Won, K., Liu, C., Naji, A., Avrahami, D., Golson, M. L., & Kaestner, K. H. (2016) · Diabetes · doi:10.2337/db16-0405 · Cited by 434 · see citing works
A statistical approach for identifying differential distributions in single-cell RNA-seq experiments
Korthauer, K. D., Chu, L.-F., Newton, M. A., Li, Y., Thomson, J., Stewart, R., & Kendziorski, C. (2016) · Genome Biology · doi:10.1186/s13059-016-1077-y · Cited by 273 · see citing works
Brennecke, P., Reyes, A., Pinto, S., Rattay, K., Nguyen, M., Küchler, R., Huber, W. K., Kyewski, B., & Steinmetz, L. M. (2015) · Nature Immunology · doi:10.1038/ni.3246 · Cited by 155 · see citing works
Smyth, G. K., & Verbyla, A. P. (1996) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1996.tb02101.x · Cited by 140 · see citing works
Delmans, M., & Hemberg, M. (2016) · BMC Bioinformatics · doi:10.1186/s12859-016-0944-6 · Cited by 111 · see citing works
Delmans, M., & Hemberg, M. (2015) · doi:10.1101/020735 · Cited by 111 · see citing works
Beyond comparisons of means: understanding changes in gene expression at the single-cell level
Vallejos, C. A., Richardson, S. T., & Marioni, J. C. (2016) · Genome biology · doi:10.1186/s13059-016-0930-3 · Cited by 94 · see citing works
Comparison of methods to detect differentially expressed genes between single-cell populations
Jaakkola, M. K., Seyednasrollah, F., Mehmood, A., & Elo, L. L. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw057 · Cited by 86 · see citing works
ROTS: reproducible RNA-seq biomarker detector—prognostic markers for clear cell renal cell cancer
Seyednasrollah, F., Rantanen, K., Jaakkola, P., & Elo, L. L. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv806 · Cited by 75 · see citing works
Mouse Embryonic Stem Cell-Derived Spheres with Distinct Neurogenic Potentials
Moliner, A., Ernfors, P., Ibáñez, C. F., & Andäng, M. (2008) · Stem Cells and Development · doi:10.1089/scd.2007.0211 · Cited by 37 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Orchestrating single-cell analysis with Bioconductor
Amezquita, R. A., Lun, A. T. L., Becht, E., Carey, V. J., Carpp, L. N., Geistlinger, L., Marini, F., Rue-Albrecht, K., Risso, D., Soneson, C., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0654-x · Cited by 945 · see citing works
Dysregulation of brain and choroid plexus cell types in severe COVID-19
Yang, A. C., Kern, F., Losada, P. M., Agam, M. R., Maat, C. A., Schmartz, G. P., Fehlmann, T., Stein, J. A., Schaum, N., Lee, D. P., et al. (2021) · Nature · doi:10.1038/s41586-021-03710-0 · Cited by 620 · see citing works
A practical solution to pseudoreplication bias in single-cell studies
Zimmerman, K. D., Espeland, M. A., & Langefeld, C. D. (2021) · Nature Communications · doi:10.1038/s41467-021-21038-1 · Cited by 362 · see citing works
Wang, T., Li, B., Nelson, C. E., & Nabavi, S. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2599-6 · Cited by 286 · see citing works
“For each generated dataset, we randomly selected 40 out of the 80 cells as one group and considered the remaining 40 cells as the other group [42].”
BackgroundMethodSimultaneous profiling of transcriptome and DNA methylome from a single cell
Hu, Y., Huang, K., An, Q., Du, G., Hu, G., Xue, J., Zhu, X., Wang, C.-Y., Xue, Z., & Fan, G. (2016) · Genome Biology · doi:10.1186/s13059-016-0950-z · Cited by 300 · see citing works
Chen, S., & Mar, J. C. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2217-z · Cited by 235 · see citing works
Single-cell RNA-seq uncovers a robust transcriptional response to morphine by glia
Avey, D., Sankararaman, S., Yim, A. K. Y., Barve, R., Milbrandt, J., & Mitra, R. D. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.08.080 · Cited by 143 · see citing works
Das, S., Rai, A., Merchant, M. L., Cave, M. C., & Rai, S. N. (2021) · Genes · doi:10.3390/genes12121947 · Cited by 35 · see citing works
Simultaneous Profiling of mRNA Transcriptome and DNA Methylome from a Single Cell
Hu, Y., An, Q., Guo, Y., Zhong, J., Fan, S., Rao, P., Liu, X., Liu, Y., & Fan, G. (2019) · Methods in Molecular Biology · doi:10.1007/978-1-4939-9240-9_21 · Cited by 26 · see citing works
Ramirez, M. D., Bui, T. N., & Katz, P. S. (2024) · Journal of Comparative Neurology · doi:10.1002/cne.25628 · Cited by 10 · see citing works
Pseudoreplication bias in single-cell studies; a practical solution
Zimmerman, K. D., Espeland, M. A., & Langefeld, C. D. (2020) · doi:10.1101/2020.01.15.906248 · Cited by 332 · see citing works
- Authors
- Dal Molin, A., Baruzzo, G., & Di Camillo, B.
- Affiliations
- University of Padua (IT)
- ORCID
- Baruzzo, Giacomo
- Venue
- Frontiers in Genetics
- Publisher
- Frontiers Media SA
- Published
- 2017-05-23
- Volume
- 8
- Pages
- 62-62
- ISSN
- 1664-8021 (ISSN-L)
- PMC
- PMC5440469
- OpenAlex
- W2617939744
- Cited by
- 107
- References
- 30
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 5 · 2023: 23 · 2022: 15 · 2021: 13 · 2020: 11 · 2019: 13 · 2018: 14 · 2017: 4 · 2016: 1
- FWCI
- 4.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Frontiers in Genetics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Padua Research Archive (University of Padova) (submittedVersion, cc-by)
- Funders
- Università degli Studi di Padova (PRAT 2010 CPDA101217)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- single-cell RNA sequencing, differential expression analysis, method comparison, single-cell transcriptomics, synthetic data, precision and recall, gene expression variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
