scRNA-seq mixology: towards better benchmarking of single cell RNA-seq analysis methods
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Single cell RNA sequencing (scRNA-seq) technology has undergone rapid development in recent years, bringing with new challenges in data processing and analysis. This has led to an explosion of tailored analysis methods for scRNA-seq data to address various biological questions. However, the current lack of gold-standard benchmark datasets makes it difficult for researchers to systematically evaluate the performance of the many methods available. Here, we designed and carried out a realistic benchmark experiment that included mixtures of single cells or ‘pseudo cells’ created by sampling admixtures of cells or RNA from up to 5 distinct cancer cell lines. Altogether we generated 14 datasets using droplet and plate-based scRNA-seq protocols, compared multiple data analysis methods in combination for tasks ranging from normalization and imputation, to clustering, trajectory analysis and data integration. Evaluation across 3,913 analyses (methods × benchmark dataset combinations) revealed pipelines suited to different types of data for different tasks. Our dataset and analysis present a comprehensive comparison framework for benchmarking most common scRNA-seq analysis tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106
Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris
The Tabula Muris Consortium, coordination, O., coordination, L., processing, O. C. A., sequencing, L. P. A., analysis, C. D., annotation, C. T., Writing group, Supplemental text writing group, & Principal investigators (2018) · Nature · doi:10.1038/s41586-018-0590-4 · Cited by 2,842
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
SC3: consensus clustering of single-cell RNA-seq data
Kiselev, V. Y., Kirschner, K., Schaub, M. T., Andrews, T., Yiu, A., Chandra, T., Natarajan, K. N., Reik, W., Barahona, M., Green, A. R., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4236 · Cited by 1,500
CEL-seq2: sensitive highly-multiplexed single-cell RNA-Seq
Hashimshony, T., Senderovich, N., Avital, G., Klochendler, A., de Leeuw, Y., Anavy, L., Gennert, D., Li, S., Livak, K. J., Rozenblatt-Rosen, O., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-0938-8 · Cited by 1,125 · cited 1×
“For the single cell designs, the 3 cell lines were mixed equally and processed by 10X Chromium, Drop-seq [31] and CEL-seq2 [13] (referred to as sc 10X, sc Drop-seq and sc CEL-seq2 respectively).”
MethodPooling across cells to normalize single-cell RNA sequencing data with many zero counts
L. Lun, A. T., Bach, K., & Marioni, J. C. (2016) · Genome Biology · doi:10.1186/s13059-016-0947-7 · Cited by 1,108
Multiplexed droplet single-cell RNA-sequencing using natural genetic variation
Kang, H. M., Subramaniam, M., Targ, S., Nguyen, M., Maliskova, L., McCarthy, E., Wan, E., Wong, S., Byrnes, L., Lanata, C. M., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4042 · Cited by 1,101
Li, H., Courtois, E. T., Sengupta, D., Tan, Y., Chen, K. H., Goh, J. J. L., Kong, S. L., Chua, C., Hon, L. K., Tan, W. S., et al. (2017) · Nature Genetics · doi:10.1038/ng.3818 · Cited by 1,080
Clustering trees: a visualization for evaluating clusterings at multiple resolutions
Zappia, L., & Oshlack, A. (2018) · GigaScience · doi:10.1093/gigascience/giy083 · Cited by 1,064
SEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983
Cited by
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
Orchestrating single-cell analysis with Bioconductor
Amezquita, R. A., Lun, A. T. L., Becht, E., Carey, V. J., Carpp, L. N., Geistlinger, L., Marini, F., Rue-Albrecht, K., Risso, D., Soneson, C., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0654-x · Cited by 945
Doublet identification in single-cell sequencing data using scDblFinder
Germain, P.-L., Lun, A., Garcia Meixide, C., Macnair, W., & Robinson, M. D. (2022) · F1000Research · doi:10.12688/f1000research.73600.2 · Cited by 684
Probabilistic cell-type assignment of single-cell RNA-seq for tumor microenvironment profiling
Zhang, A. W., O’Flanagan, C., Chavez, E. A., Lim, J. L. P., Ceglia, N., McPherson, A., Wiens, M., Walters, P., Chan, T., Hewitson, B., et al. (2019) · Nature Methods · doi:10.1038/s41592-019-0529-1 · Cited by 375
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220
Self-citation
Metadata
- Authors
- Tian, L., Dong, X., Freytag, S., Lê Cao, K.-A., Su, S., JalalAbadi, A., Amann-Zalcenstein, D., Weber, T. S., Seidi, A., Jabbari, J. S., Naik, S. H., & Ritchie, M. E.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-10-03
- OpenAlex
- W2895662121
- Cited by
- 11
- References
- 25
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.