Benchmarking methods for detecting differential states between conditions from multi-subject single-cell RNA-seq data
Open accessPublished version of a preprint
Abstract
Single-cell RNA-sequencing (scRNA-seq) enables researchers to quantify transcriptomes of thousands of cells simultaneously and study transcriptomic changes between cells. scRNA-seq datasets increasingly include multisubject, multicondition experiments to investigate cell-type-specific differential states (DS) between conditions. This can be performed by first identifying the cell types in all the subjects and then by performing a DS analysis between the conditions within each cell type. Naïve single-cell DS analysis methods that treat cells statistically independent are subject to false positives in the presence of variation between biological replicates, an issue known as the pseudoreplicate bias. While several methods have already been introduced to carry out the statistical testing in multisubject scRNA-seq analysis, comparisons that include all these methods are currently lacking. Here, we performed a comprehensive comparison of 18 methods for the identification of DS changes between conditions from multisubject scRNA-seq data. Our results suggest that the pseudobulk methods performed generally best. Both pseudobulks and mixed models that model the subjects as a random effect were superior compared with the naïve single-cell methods that do not model the subjects in any way. While the naïve models achieved higher sensitivity than the pseudobulk methods and the mixed models, they were subject to a high number of false positives. In addition, accounting for subjects through latent variable modeling did not improve the performance of the naïve methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pseudoreplication leads to an inflated false discovery rate because differential gene expression methods do not account for the inherent correlation of replicates.
“Pseudoreplication leads to an inflated false discovery rate (FDR) as DGE methods do not account for the inherent correlation of replicates (cells from the same individual)86–88.”
From Best practices for single-cell analysis across modalities (2023) · cites this paper as [88]
Pseudobulk methods with sum aggregation and mixed models such as MAST with random effect setting were superior to naive methods such as Wilcoxon rank-sum test.
“Generally, pseudobulk methods with sum aggregation and mixed models such as MAST with random effect setting were found to be superior to naive methods, such as the popular Wilcoxon rank-sum test, which does not account for within-sample correlation88.”
From Best practices for single-cell analysis across modalities (2023) · cites this paper as [88]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106 · see citing works
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
Robinson, M. D., McCarthy, D. J., & Smyth, G. K. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btp616 · Cited by 41,364 · see citing works
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458 · see citing works
Integrated analysis of multimodal single-cell data
Hao, Y., Hao, S., Andersen-Nissen, E., Mauck, W. M., Zheng, S., Butler, A., Lee, M. J., Wilk, A. J., Darby, C., Zagar, M., et al. (2020) · doi:10.1101/2020.10.12.335331 · Cited by 15,508 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
Integrating single-cell transcriptomic data across different conditions, technologies, and species
Butler, A., Hoffman, P., Smibert, P., Papalexi, E., & Satija, R. (2018) · Nature Biotechnology · doi:10.1038/nbt.4096 · Cited by 13,174 · see citing works
Adjusting batch effects in microarray expression data using empirical Bayes methods
Johnson, W. E., Li, C., & Rabinovic, A. (2007) · Biostatistics · doi:10.1093/biostatistics/kxj037 · Cited by 8,658 · see citing works
A scaling normalization method for differential expression analysis of RNA-seq data
Robinson, M. D., & Oshlack, A. (2010) · Genome Biology · doi:10.1186/gb-2010-11-3-r25 · Cited by 7,876 · see citing works
Massively parallel digital transcriptional profiling of single cells
Zheng, G. X. Y., Terry, J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017) · Nature Communications · doi:10.1038/ncomms14049 · Cited by 7,353 · see citing works
voom: precision weights unlock linear model analysis tools for RNA-seq read counts
Law, C. W., Chen, Y., Shi, W., & Smyth, G. K. (2014) · Genome Biology · doi:10.1186/gb-2014-15-2-r29 · Cited by 6,155 · see citing works
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
Finak, G., McDavid, A., Yajima, M., Deng, J., Gersuk, V., Shalek, A. K., Slichter, C. K., Miller, H. W., McElrath, M. J., Prlic, M., et al. (2015) · Genome Biology · doi:10.1186/s13059-015-0844-5 · Cited by 3,159 · see citing works
Benchmarking atlas-level data integration in single-cell genomics
Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Mueller, M. F., Strobl, D. C., Zappia, L., Dugas, M., Colomé-Tatché, M., et al. (2022) · Nature Methods · doi:10.1038/s41592-021-01336-8 · Cited by 1,438 · see citing works
Multiplexed droplet single-cell RNA-sequencing using natural genetic variation
Kang, H. M., Subramaniam, M., Targ, S., Nguyen, M., Maliskova, L., McCarthy, E., Wan, E., Wong, S., Byrnes, L., Lanata, C. M., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4042 · Cited by 1,339 · see citing works
Confronting false discoveries in single-cell differential expression
Squair, J. W., Gautier, M., Kathe, C., Anderson, M. A., James, N. D., Hutson, T. H., Hudelle, R., Qaiser, T., Matson, K. J. E., Barraud, Q., et al. (2021) · doi:10.1101/2021.03.12.435024 · Cited by 1,017 · cited 1× · see citing works
“[10] compared naïve methods, pseudobulk methods and one mixed-model method (muscat_MM).”
MethodBias, robustness and scalability in single-cell differential expression analysis
Soneson, C., & Robinson, M. D. (2018) · Nature Methods · doi:10.1038/nmeth.4612 · Cited by 770 · see citing works
Classification of low quality cells from single-cell RNA-seq data
Ilicic, T., Kim, J. K., Kolodziejczyk, A. A., Bagger, F. O., McCarthy, D. J., Marioni, J. C., & Teichmann, S. A. (2016) · Genome Biology · doi:10.1186/s13059-016-0888-1 · Cited by 693 · see citing works
Crowell, H. L., Soneson, C., Germain, P.-L., Calini, D., Collin, L., Raposo, C., Malhotra, D., & Robinson, M. D. (2020) · Nature Communications · doi:10.1038/s41467-020-19894-4 · Cited by 515 · see citing works
A practical solution to pseudoreplication bias in single-cell studies
Zimmerman, K. D., Espeland, M. A., & Langefeld, C. D. (2021) · Nature Communications · doi:10.1038/s41467-021-21038-1 · Cited by 362 · cited 5× · see citing works
“The comparison did not consider the recently introduced MAST model (MAST_RE) [8] or NEBULA, which is another recently introduced mixed model specifically designed for the DS analysis of multisubject scRNA-seq data [16].”
BackgroundMethodInfluentialDroplet scRNA-seq is not zero-inflated
Svensson, V. (2019) · doi:10.1101/582064 · Cited by 356 · see citing works
A statistical approach for identifying differential distributions in single-cell RNA-seq experiments
Korthauer, K. D., Chu, L.-F., Newton, M. A., Li, Y., Thomson, J., Stewart, R., & Kendziorski, C. (2016) · Genome Biology · doi:10.1186/s13059-016-1077-y · Cited by 273 · see citing works
A systematic evaluation of single cell RNA-seq analysis pipelines
Vieth, B., Parekh, S., Ziegenhain, C., Enard, W., & Hellmann, I. (2019) · Nature Communications · doi:10.1038/s41467-019-12266-7 · Cited by 265 · see citing works
Time-resolved systems immunology reveals a late juncture linked to fatal COVID-19
Liu, C., Martins, A. J., Lau, W. W., Rachmaninoff, N., Chen, J., Imberti, L., Mostaghimi, D., Fink, D. L., Burbelo, P. D., Dobbs, K., et al. (2021) · Cell · doi:10.1016/j.cell.2021.02.018 · Cited by 209 · see citing works
He, L., Davila-Velderrain, J., Sumida, T. S., Hafler, D. A., Kellis, M., & Kulminski, A. M. (2021) · Communications Biology · doi:10.1038/s42003-021-02146-6 · Cited by 181 · see citing works
Performance Assessment and Selection of Normalization Procedures for Single-Cell RNA-Seq
Cole, M. B., Risso, D., Wagner, A., DeTomaso, D., Ngai, J., Purdom, E., Dudoit, S., & Yosef, N. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.010 · Cited by 173 · see citing works
ROTS: An R package for reproducibility-optimized statistical testing
Suomi, T., Seyednasrollah, F. S., Jaakkola, M. K., Faux, T., & Elo, L. L. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005562 · Cited by 149 · see citing works
Self-citationComparison of methods to detect differentially expressed genes between single-cell populations
Jaakkola, M. K., Seyednasrollah, F., Mehmood, A., & Elo, L. L. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw057 · Cited by 86 · see citing works
Self-citationKallionpää, H., Somani, J., Tuomela, S., Ullah, U., de Albuquerque, R., Lönnberg, T., Komsi, E., Siljander, H., Honkanen, J. K., Härkönen, T., et al. (2019) · Diabetes · doi:10.2337/db19-0287 · Cited by 63 · see citing works
AAV1 is the optimal viral vector for optogenetic experiments in pigeons (Columba livia)
Rook, N., Tuff, J. M., Isparta, S., Masseck, O. A., Herlitze, S., Güntürkün, O., & Pusch, R. (2021) · Communications Biology · doi:10.1038/s42003-020-01595-9 · Cited by 36 · see citing works
Ganna, A., Lee, D., Ingelsson, E., & Yudi Pawitan, Y. (2014) · Briefings in Bioinformatics · doi:10.1093/bib/bbu033 · Cited by 32 · see citing works
Thurman, A. L., Ratcliff, J. A., Chimenti, M. S., & Pezzulo, A. A. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab337 · Cited by 27 · see citing works
distinct: A novel approach to differential distribution analyses
Tiberi, S., Crowell, H. L., Samartsidis, P., Weber, L. M., & Robinson, M. D. (2023) · The Annals of Applied Statistics · doi:10.1214/22-aoas1689 · Cited by 25 · see citing works
distinct : a novel approach to differential distribution analysesPreprint
Tiberi, S., Crowell, H. L., Samartsidis, P., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.11.24.394213 · Cited by 11 · see citing works
2024-P: Protective Role of Lipocalin 2 in High-Fat Diet-Induced Gut Microbiota Dysbiosis
Qiu, X., Macchietto, M. G., Shen, S. S., et al. (2019) · Diabetes · doi:10.2337/db19-2024-p · Cited by 7 · see citing works
distinct: a novel approach to differential distribution analyses
Tiberi, S., Crowell, H. L., Weber, L. M., et al. (2020) · doi:10.7490/f1000research.1118440.1 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Best practices for single-cell analysis across modalities
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. (2023) · Nature Reviews Genetics · doi:10.1038/s41576-023-00586-w · Cited by 1,051 · see citing works
“3).”
- Authors
- Junttila, S., Smolander, J., & Elo, L. L.
- Affiliations
- Åbo Akademi University (FI)
- Corresponding author
- Elo, Laura L
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-07-25
- Volume
- 23
- Issue
- 5
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- PMC
- PMC9487674
- OpenAlex
- W4288042632
- Cited by
- 109
- References
- 36
- Citations by year
- 2026: 39 · 2025: 21 · 2024: 30 · 2023: 16 · 2022: 2 · 2020: 1
- FWCI
- 6.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Briefings in Bioinformatics (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UTUPub (University of Turku) (submittedVersion)
- Funders
- European Research Council (677943)
- European Union’s Horizon 2020 (955321)
- Academy of Finland (310561, 314443, 329278, 335434, 335611, 341342)
- Sigrid Juselius Foundation
- University of Turku
- Åbo Akademi University
- Turku Graduate School
- Biocenter Finland
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- mixed models, random effects, benchmarking, false positives, latent variable models, method comparison
- MeSH terms
- RNA-Seq, Humans, RNA, Sequence Analysis, RNA, Benchmarking, Gene Expression Profiling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
