scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Single cell RNA sequencing (scRNA-seq) technology has undergone rapid development in recent years, bringing with new challenges in data processing and analysis. This has led to an explosion of tailored analysis methods for scRNA-seq data to address various biological questions. However, the current lack of gold-standard benchmark datasets makes it difficult for researchers to systematically evaluate the performance of the many methods available. Here, we designed and carried out a realistic benchmark experiment that included mixtures of single cells or ‘pseudo cells’ created by sampling admixtures of cells or RNA from up to 5 distinct cancer cell lines. Altogether we generated 14 datasets using droplet and plate-based scRNA-seq protocols, compared multiple data analysis methods in combination for tasks ranging from normalization and imputation, to clustering, trajectory analysis and data integration. Evaluation across 3,913 analyses (methods × benchmark dataset combinations) revealed pipelines suited to different types of data for different tasks. Our dataset and analysis present a comprehensive comparison framework for benchmarking most common scRNA-seq analysis tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106
Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris
The Tabula Muris Consortium, coordination, O., coordination, L., processing, O. C. A., sequencing, L. P. A., analysis, C. D., annotation, C. T., Writing group, Supplemental text writing group, & Principal investigators (2018) · Nature · doi:10.1038/s41586-018-0590-4 · Cited by 2,842
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
SC3: consensus clustering of single-cell RNA-seq data
Kiselev, V. Y., Kirschner, K., Schaub, M. T., Andrews, T., Yiu, A., Chandra, T., Natarajan, K. N., Reik, W., Barahona, M., Green, A. R., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4236 · Cited by 1,500
CEL-seq2: sensitive highly-multiplexed single-cell RNA-Seq
Hashimshony, T., Senderovich, N., Avital, G., Klochendler, A., de Leeuw, Y., Anavy, L., Gennert, D., Li, S., Livak, K. J., Rozenblatt-Rosen, O., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-0938-8 · Cited by 1,125 · cited 1×
“For the single cell designs, the 3 cell lines were mixed equally and processed by 10X Chromium, Drop-seq [31] and CEL-seq2 [13] (referred to as sc 10X, sc Drop-seq and sc CEL-seq2 respectively).”
MethodPooling across cells to normalize single-cell RNA sequencing data with many zero counts
L. Lun, A. T., Bach, K., & Marioni, J. C. (2016) · Genome Biology · doi:10.1186/s13059-016-0947-7 · Cited by 1,108
Multiplexed droplet single-cell RNA-sequencing using natural genetic variation
Kang, H. M., Subramaniam, M., Targ, S., Nguyen, M., Maliskova, L., McCarthy, E., Wan, E., Wong, S., Byrnes, L., Lanata, C. M., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4042 · Cited by 1,101
Li, H., Courtois, E. T., Sengupta, D., Tan, Y., Chen, K. H., Goh, J. J. L., Kong, S. L., Chua, C., Hon, L. K., Tan, W. S., et al. (2017) · Nature Genetics · doi:10.1038/ng.3818 · Cited by 1,080
Clustering trees: a visualization for evaluating clusterings at multiple resolutions
Zappia, L., & Oshlack, A. (2018) · GigaScience · doi:10.1093/gigascience/giy083 · Cited by 1,064
SEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983
Bias, robustness and scalability in single-cell differential expression analysis
Soneson, C., & Robinson, M. D. (2018) · Nature Methods · doi:10.1038/nmeth.4612 · Cited by 770
Validation of noise models for single-cell transcriptomics
Grün, D., Kester, L., & van Oudenaarden, A. (2014) · Nature Methods · doi:10.1038/nmeth.2930 · Cited by 721
A general and flexible method for signal extraction from single-cell RNA-seq data
Risso, D., Perraudeau, F., Gribkova, S., Dudoit, S., & Vert, J.-P. (2018) · Nature Communications · doi:10.1038/s41467-017-02554-5 · Cited by 718 · cited 2×
“Since the single cell and RNA mixture data were generated using multiple protocols, we used these datasets to compare these state-of-art methods including MNN [12], Scanorama [14], scMerge [28], ZINB-WaVE [37], Seurat [43] and MINT [40].”
MethodSynthetic spike-in standards for RNA-seq experiments
Jiang, L., Schlesinger, F., Davis, C. A., Zhang, Y., Li, R., Salit, M., Gingeras, T. R., & Oliver, B. (2011) · Genome Research · doi:10.1101/gr.121095.111 · Cited by 653 · cited 1×
“The first uses small collections of exogenous spike-in controls (such as ERCCs [19]) that vary in expression in a predictable way, which have been widely adopted in scRNA-seq studies [47].”
BackgroundPower analysis of single-cell RNA-sequencing experiments
Svensson, V., Natarajan, K. N., Ly, L.-H., Miragaia, R. J., Labalette, C., Macaulay, I. C., Cvejic, A., & Teichmann, S. A. (2017) · Nature Methods · doi:10.1038/nmeth.4220 · Cited by 568
A test metric for assessing single-cell RNA-seq batch correction
Büttner, M., Miao, Z., Wolf, F. A., Teichmann, S. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-018-0254-1 · Cited by 511
FateID infers cell fate bias in multipotent progenitors from single-cell RNA-seq data
Herman, J. S., Sagar, & Grün, D. (2018) · Nature Methods · doi:10.1038/nmeth.4662 · Cited by 363
Batch effects and the effective design of single-cell gene expression studies
Tung, P.-Y., Blischak, J. D., Hsiao, C. J., Knowles, D. A., Burnett, J. E., Pritchard, J. K., & Gilad, Y. (2017) · Scientific Reports · doi:10.1038/srep39921 · Cited by 356 · cited 2×
“Mixture designs, in which RNA or cells are mixed in different proportions to generate biological heterogeneity with in-built truth have been successfully used to benchmark microarray [5], RNA-seq [44] and scRNA-seq data [49].”
BackgroundBASiCS: Bayesian Analysis of Single-Cell Sequencing Data
Vallejos, C. A., Marioni, J. C., & Richardson, S. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004333 · Cited by 302 · cited 2×
“…12 normalization and imputation methods, with methods designed for bulk RNA-seq such as TMM [39], CPM [38] and DESeq [29], and methods designed for scRNA-seq, including ZINB-WaVE [37], scone [4], kNN smooth [52], BASiCS [50], SCnorm [1], Linnorm [55], scran [30], SAVER [16] and DrImpute [23].”
MethodA systematic performance evaluation of clustering methods for single-cell RNA-seq data
Duò, A., Robinson, M. D., & Soneson, C. (2018) · F1000Research · doi:10.12688/f1000research.15666.2 · Cited by 296
SCnorm: robust normalization of single-cell RNA-seq data
Bacher, R., Chu, L.-F., Leng, N., Gasch, A. P., Thomson, J. A., Stewart, R. M., Newton, M., & Kendziorski, C. (2017) · Nature Methods · doi:10.1038/nmeth.4263 · Cited by 282
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220
Self-citationPerformance Assessment and Selection of Normalization Procedures for Single-Cell RNA-Seq
Cole, M. B., Risso, D., Wagner, A., DeTomaso, D., Ngai, J., Purdom, E., Dudoit, S., & Yosef, N. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.010 · Cited by 173
False signals induced by single-cell imputation
Andrews, T. S., & Hemberg, M. (2018) · F1000Research · doi:10.12688/f1000research.16613.1 · Cited by 142
Linnorm: improved statistical analysis for single cell RNA-seq expression data
Yip, S. H., Wang, P., Kocher, J.-P. A., Sham, P. C., & Wang, J. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx828 · Cited by 121
Cited by
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
Orchestrating single-cell analysis with Bioconductor
Amezquita, R. A., Lun, A. T. L., Becht, E., Carey, V. J., Carpp, L. N., Geistlinger, L., Marini, F., Rue-Albrecht, K., Risso, D., Soneson, C., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0654-x · Cited by 945
Doublet identification in single-cell sequencing data using scDblFinder
Germain, P.-L., Lun, A., Garcia Meixide, C., Macnair, W., & Robinson, M. D. (2022) · F1000Research · doi:10.12688/f1000research.73600.2 · Cited by 684
Probabilistic cell-type assignment of single-cell RNA-seq for tumor microenvironment profiling
Zhang, A. W., O’Flanagan, C., Chavez, E. A., Lim, J. L. P., Ceglia, N., McPherson, A., Wiens, M., Walters, P., Chan, T., Hewitson, B., et al. (2019) · Nature Methods · doi:10.1038/s41592-019-0529-1 · Cited by 375
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220
Self-citation
Metadata
- Authors
- Tian, L., Dong, X., Freytag, S., Lê Cao, K.-A., Su, S., JalalAbadi, A., Amann-Zalcenstein, D., Weber, T. S., Seidi, A., Jabbari, J. S., Naik, S. H., & Ritchie, M. E.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-10-03
- OpenAlex
- W2895662121
- Cited by
- 11
- References
- 25
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.