The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
A multicenter study benchmarking single-cell RNA sequencing technologies using reference samples
Open accessPublished version of a preprint
Abstract
Comparing diverse single-cell RNA sequencing (scRNA-seq) datasets generated by different technologies and in different laboratories remains a major challenge. Here we address the need for guidance in choosing algorithms leading to accurate biological interpretations of varied data types acquired with different platforms. Using two well-characterized cellular reference samples (breast cancer cells and B cells), captured either separately or in mixtures, we compared different scRNA-seq platforms and several preprocessing, normalization and batch-effect correction methods at multiple centers. Although preprocessing and normalization contributed to variability in gene detection and cell classification, batch-effect correction was by far the most important factor in correctly classifying the cells. Moreover, scRNA-seq dataset characteristics (for example, sample and cellular heterogeneity and platform used) were critical in determining the optimal bioinformatic method. However, reproducibility across centers and platforms was high when appropriate bioinformatic methods were applied. Our findings offer practical guidance for optimizing platform and software selection when designing an scRNA-seq study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent benchmark studies indicate that for simpler integration problems with relatively low biological complexity and relatively small batch effects in scRNA-Seq datasets, linear methods outperform nonlinear, more complex methods136,137, while for complicated integration tasks with large biological complexity and bigger batch effects, nonlinear methods such as scVI/scanVI outperform others138.”
From A high-resolution transcriptomic and spatial atlas of cell types in the whole mouse brain (2023) · cites this paper as [136]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Trimmomatic: a flexible trimmer for Illumina sequence data
Bolger, A. M., Lohse, M., & Usadel, B. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu170 · Cited by 65,548
The Sequence Alignment/Map format and SAMtools
Li, H., Handsaker, B., Wysoker, A., Fennell, T., Ruan, J., Homer, N., Marth, G., Abecasis, G., Durbin, R., & 1000 Genome Project Data Processing Subgroup (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp352 · Cited by 62,041
STAR: ultrafast universal RNA-seq aligner
Dobin, A., Davis, C. A., Schlesinger, F., Drenkow, J., Zaleski, C., Jha, S., Batut, P., Chaisson, M., & Gingeras, T. R. (2013) · Bioinformatics · doi:10.1093/bioinformatics/bts635 · Cited by 51,709
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458
Cutadapt removes adapter sequences from high-throughput sequencing reads
Martin, M. (2011) · EMBnet.journal · doi:10.14806/ej.17.1.200 · Cited by 33,889
featureCounts: an efficient general purpose program for assigning sequence reads to genomic features
Liao, Y., Smyth, G. K., & Shi, W. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btt656 · Cited by 27,219
RSEM: accurate transcript quantification from RNA-Seq data with or without a reference genome
Li, B., & Dewey, C. N. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-323 · Cited by 22,179
Comprehensive Integration of Single-Cell Data
Stuart, T., Butler, A., Hoffman, P., Hafemeister, C., Papalexi, E., Mauck, W. M., Hao, Y., Stoeckius, M., Smibert, P., & Satija, R. (2019) · Cell · doi:10.1016/j.cell.2019.05.031 · Cited by 15,453
Integrating single-cell transcriptomic data across different conditions, technologies, and species
Butler, A., Hoffman, P., Smibert, P., Papalexi, E., & Satija, R. (2018) · Nature Biotechnology · doi:10.1038/nbt.4096 · Cited by 13,174
Fast, sensitive and accurate integration of single-cell data with Harmony
Korsunsky, I., Millard, N., Fan, J., Slowikowski, K., Zhang, F., Wei, K., Baglaenko, Y., Brenner, M., Loh, P.-R., & Raychaudhuri, S. (2019) · Nature Methods · doi:10.1038/s41592-019-0619-0 · Cited by 10,488
Near-optimal probabilistic RNA-seq quantification
Bray, N. L., Pimentel, H., Melsted, P., & Pachter, L. (2016) · Nature Biotechnology · doi:10.1038/nbt.3519 · Cited by 9,723
Trapnell, C., Cacchiarelli, D., Grimsby, J., Pokharel, P., Li, S., Morse, M., Lennon, N. J., Livak, K. J., Mikkelsen, T. S., & Rinn, J. L. (2014) · Nature Biotechnology · doi:10.1038/nbt.2859 · Cited by 7,314
Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets.
Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M., Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.002 · Cited by 7,110
Leek, J. T., Johnson, W. E., Parker, H. S., Jaffe, A. E., & Storey, J. D. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts034 · Cited by 6,231
Dimensionality reduction for visualizing single-cell data using UMAP
Becht, E., McInnes, L., Healy, J., Dutertre, C.-A., Kwok, I. W. H., Ng, L. G., Ginhoux, F., & Newell, E. W. (2019) · Nature Biotechnology · doi:10.1038/nbt.4314 · Cited by 5,358
Hafemeister, C., & Satija, R. (2019) · Genome Biology · doi:10.1186/s13059-019-1874-1 · Cited by 4,479
Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells
Klein, A. M., Mazutis, L., Akartuna, I., Tallapragada, N., Veres, A., Li, V., Peshkin, L., Weitz, D. A., & Kirschner, M. W. (2015) · Cell · doi:10.1016/j.cell.2015.04.044 · Cited by 3,356
Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors
Haghverdi, L., Lun, A. T. L., Morgan, M. D., & Marioni, J. C. (2018) · Nature Biotechnology · doi:10.1038/nbt.4091 · Cited by 2,416
Smith, T. S., Heger, A., & Sudbery, I. M. (2017) · Genome Research · doi:10.1101/gr.209601.116 · Cited by 2,072
Normalization of RNA-seq data using factor analysis of control genes or samples
Risso, D., Ngai, J., Speed, T. P., & Dudoit, S. (2014) · Nature Biotechnology · doi:10.1038/nbt.2931 · Cited by 2,052
Single-cell mRNA quantification and differential analysis with Census
Qiu, X., Hill, A. J., Packer, J. S., Lin, D., Ma, Y.-A., & Trapnell, C. (2017) · Nature Methods · doi:10.1038/nmeth.4150 · Cited by 1,904
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
Comparative Analysis of Single-Cell RNA Sequencing Methods
Ziegenhain, C., Vieth, B., Parekh, S., Reinius, B., Guillaumet-Adkins, A., Smets, M., Leonhardt, H., Heyn, H., Hellmann, I., & Enard, W. (2017) · Molecular Cell · doi:10.1016/j.molcel.2017.01.023 · Cited by 1,494
Buettner, F., Natarajan, K. N., Casale, F. P., Proserpio, V., Scialdone, A., Theis, F. J., Teichmann, S. A., Marioni, J. C., & Stegle, O. (2015) · Nature Biotechnology · doi:10.1038/nbt.3102 · Cited by 1,136
A benchmark of batch-effect correction methods for single-cell RNA sequencing data
Tran, H. T. N., Ang, K. S., Chevrier, M., Zhang, X., Lee, N. Y. S., Goh, M., & Chen, J. (2020) · Genome Biology · doi:10.1186/s13059-019-1850-9 · Cited by 1,113
Pooling across cells to normalize single-cell RNA sequencing data with many zero counts
L. Lun, A. T., Bach, K., & Marioni, J. C. (2016) · Genome Biology · doi:10.1186/s13059-016-0947-7 · Cited by 1,108
Multiplexed droplet single-cell RNA-sequencing using natural genetic variation
Kang, H. M., Subramaniam, M., Targ, S., Nguyen, M., Maliskova, L., McCarthy, E., Wan, E., Wong, S., Byrnes, L., Lanata, C. M., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4042 · Cited by 1,101
BBKNN: fast batch alignment of single cell transcriptomes
Polański, K., Young, M. D., Miao, Z., Meyer, K. B., Teichmann, S. A., & Park, J.-E. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz625 · Cited by 932
Seq-Well: portable, low-cost RNA sequencing of single cells at high throughput
Gierahn, T. M., Wadsworth, M. H., Hughes, T. K., Bryson, B. D., Butler, A., Satija, R., Fortune, S. M., Love, J. C., & Shalek, A. K. (2017) · Nature Methods · doi:10.1038/nmeth.4179 · Cited by 882
Efficient integration of heterogeneous single-cell transcriptomes using Scanorama
Hie, B., Bryson, B. D., & Berger, B. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0113-3 · Cited by 876
Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls
Zook, J. M., Chapman, B., Wang, J., Mittelman, D., Hofmann, O., Hide, W., & Salit, M. (2014) · Nature Biotechnology · doi:10.1038/nbt.2835 · Cited by 800
Missing data and technical variability in single-cell RNA-sequencing experiments
Hicks, S. C., Townes, F. W., Teng, M., & Irizarry, R. A. (2018) · Biostatistics · doi:10.1093/biostatistics/kxx053 · Cited by 635
Power analysis of single-cell RNA-sequencing experiments
Svensson, V., Natarajan, K. N., Ly, L.-H., Miragaia, R. J., Labalette, C., Macaulay, I. C., Cvejic, A., & Teichmann, S. A. (2017) · Nature Methods · doi:10.1038/nmeth.4220 · Cited by 568
Zook, J. M., Chapman, B., Wang, J. J., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.4661 · Cited by 551
A test metric for assessing single-cell RNA-seq batch correction
Büttner, M., Miao, Z., Wolf, F. A., Teichmann, S. A., & Theis, F. J. (2019) · Nature Methods · doi:10.1038/s41592-018-0254-1 · Cited by 511
Benchmarking single-cell RNA-sequencing protocols for cell atlas projects
Mereu, E., Lafzi, A., Moutinho, C., Ziegenhain, C., McCarthy, D. J., Álvarez-Varela, A., Batlle, E., Sagar, Grün, D., Lau, J. K., et al. (2020) · Nature Biotechnology · doi:10.1038/s41587-020-0469-4 · Cited by 499
Comparative Analysis of Droplet-Based Ultra-High-Throughput Single-Cell RNA-Seq Systems
Zhang, X., Li, T., Liu, F., Chen, Y., Yao, J., Li, Z., Huang, Y., & Wang, J. (2018) · Molecular Cell · doi:10.1016/j.molcel.2018.10.020 · Cited by 395
zUMIs - A fast and flexible pipeline to process RNA sequencing data with UMIs
Parekh, S., Ziegenhain, C., Vieth, B., Enard, W., & Hellmann, I. (2018) · GigaScience · doi:10.1093/gigascience/giy059 · Cited by 364
Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments
Tian, L., Dong, X., Freytag, S., Cao, K. L., Su, S., JalalAbadi, A., Amann‐Zalcenstein, D., Weber, T. S., Seidi, A., Jabbari, J. S., et al. (2019) · Nature Methods · doi:10.1038/s41592-019-0425-8 · Cited by 355
Gazdar, A. F., Kurvari, V., Virmani, A. K., Gollahon, L., Sakaguchi, M., Westerfield, M., Kodagoda, D., Stasny, V., Cunningham, H. T., Wistuba, I. I., et al. (1998) · International Journal of Cancer · doi:10.1002/(sici)1097-0215(19981209)78:6 · Cited by 336
Cell-surface CD74 initiates a signaling cascade leading to cell proliferation and survival
Starlets, D., Gore, Y., Binsky, I., Haran, M. Z., Harpaz, N., Shvidel, L., Becker-Herman, S., Berrébi, A., & Shachar, I. (2006) · Blood · doi:10.1182/blood-2005-11-4334 · Cited by 305
Advances in Human B Cell Phenotypic Profiling
Kaminski, D. A., Wei, C., Yu, Q., Rosenberg, A. F., & Sanz, I. (2012) · Frontiers in Immunology · doi:10.3389/fimmu.2012.00302 · Cited by 291
SCnorm: robust normalization of single-cell RNA-seq data
Bacher, R., Chu, L.-F., Leng, N., Gasch, A. P., Thomson, J. A., Stewart, R. M., Newton, M., & Kendziorski, C. (2017) · Nature Methods · doi:10.1038/nmeth.4263 · Cited by 282
Cumulus provides cloud-based data analysis for large-scale single-cell and single-nucleus RNA-seq
Li, B., Gould, J., Yang, Y., Sarkizova, S., Tabaka, M., Ashenberg, O., Rosen, Y., Slyper, M., Kowalczyk, M. S., Villani, A., et al. (2020) · Nature Methods · doi:10.1038/s41592-020-0905-x · Cited by 243
Cell fixation and preservation for droplet-based single-cell transcriptomics
Alles, J., Karaiskos, N., Praktiknjo, S. D., Grosswendt, S., Wahle, P., Ruffault, P.-L., Ayoub, S., Schreyer, L., Boltengagen, A., Birchmeier, C., et al. (2017) · BMC Biology · doi:10.1186/s12915-017-0383-5 · Cited by 218
Evaluation of tools for highly variable gene discovery from single-cell RNA-seq data
Yip, S. H., Sham, P. C., & Wang, J. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bby011 · Cited by 201
Toward best practice in cancer mutation detection with whole-genome and whole-exome sequencing
Xiao, W., Ren, L., Chen, Z., Fang, L. T., Zhao, Y., Lack, J., Guan, M., Zhu, B., Jaeger, E. B., Kerrigan, L., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00994-5 · Cited by 188
Self-citationPerformance Assessment and Selection of Normalization Procedures for Single-Cell RNA-Seq
Cole, M. B., Risso, D., Wagner, A., DeTomaso, D., Ngai, J., Purdom, E., Dudoit, S., & Yosef, N. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.010 · Cited by 173
Zhang, J., Späth, S. S., Marjani, S. L., Zhang, W., & Pan, X. (2018) · Precision Clinical Medicine · doi:10.1093/pcmedi/pby007 · Cited by 137
Linnorm: improved statistical analysis for single cell RNA-seq expression data
Yip, S. H., Wang, P., Kocher, J.-P. A., Sham, P. C., & Wang, J. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx828 · Cited by 121
Determining sequencing depth in a single-cell RNA-seq experiment
Zhang, M. J., Ntranos, V., & Tse, D. (2020) · Nature Communications · doi:10.1038/s41467-020-14482-y · Cited by 112
A multi-center cross-platform single-cell RNA sequencing reference dataset
Chen, X., Yang, Z., Chen, W., Zhao, Y., Farmer, A. A., Tran, B. Q., Furtak, V. A., Moos, M. C., Xiao, W., & Wang, C. (2021) · Scientific Data · doi:10.1038/s41597-021-00809-x · Cited by 28
Self-citationSingle-Cell Sequencing for Drug Discovery and Drug Development
Wu, H., Wang, C., & Wu, S. (2017) · Current Topics in Medicinal Chemistry · doi:10.2174/1568026617666161116145358 · Cited by 21
Self-citationThe Somatic Mutation Working Group of the SEQC-II Consortium (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/626440 · Cited by 12
Single-Cell Sequencing Technologies for Cardiac Stem Cell Studies
Liu, T., Wu, H., Wu, S., & Wang, C. (2017) · Stem Cells and Development · doi:10.1089/scd.2017.0050 · Cited by 9
Self-citationA Multi-center Cross-platform Single-cell RNA Sequencing Reference DatasetPreprint
Chen, X., Yang, Z., Chen, W., Zhao, Y., Farmer, A. A., Tran, B. Q., Furtak, V. A., Moos, M. C., Xiao, W., & Wang, C. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.09.20.305474 · Cited by 3
Self-citation
Cited by
Sorted by citations
Benchmarking atlas-level data integration in single-cell genomics
Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Mueller, M. F., Strobl, D. C., Zappia, L., Dugas, M., Colomé-Tatché, M., et al. (2022) · Nature Methods · doi:10.1038/s41592-021-01336-8 · Cited by 1,438
A high-resolution transcriptomic and spatial atlas of cell types in the whole mouse brain
Yao, Z., van Velthoven, C. T. J., Kunst, M., Zhang, M., McMillen, D., Lee, C., Jung, W., Goldy, J., Abdelhak, A., Aitken, M., et al. (2023) · Nature · doi:10.1038/s41586-023-06812-z · Cited by 1,203
Statistics or biology: the zero-inflation controversy about scRNA-seq data
Jiang, R., Sun, T., Song, D., & Li, J. J. (2022) · Genome Biology · doi:10.1186/s13059-022-02601-5 · Cited by 614
Self-citationTietscher, S., Wagner, J., Anzeneder, T., Langwieder, C., Rees, M., Sobottka, B., de Souza, N., & Bodenmiller, B. (2023) · Nature Communications · doi:10.1038/s41467-022-35238-w · Cited by 233
Assessing and mitigating batch effects in large-scale omics studies
Yu, Y., Mai, Y., Zheng, Y., & Shi, L. M. (2024) · Genome biology · doi:10.1186/s13059-024-03401-9 · Cited by 254
Metadata
- Authors
- Chen, W., Zhao, Y., Chen, X., Yang, Z., Xu, X., Bi, Y., Chen, V., Li, J., Choi, H., Ernest, B., Tran, B., Mehta, M., Kumar, P., Farmer, A., Mir, A., Mehra, U. A., Li, J.-L., Moos, M., Xiao, W., & Wang, C.
- Affiliations
- Loma Linda University (US)
- Frederick National Laboratory for Cancer Research (US)
- National Institute of Environmental Health Sciences (US)
- AbbVie (United States) (US)
- Takara (United States) (US)
- Center for Biologics Evaluation and Research (US)
- United States Food and Drug Administration (US)
- Corresponding author
- Xiao, Wenming
- Wang, Charles
- ORCID
- Venue
- Nature Biotechnology
- Published
- 2020-12-21
- Volume
- 39
- Issue
- 9
- Pages
- 1103-1114
- ISSN
- 0733-222X (ISSN-L)
- 1087-0156
- 1546-1696
- PMC
- PMC11245320
- OpenAlex
- W3117537317
- Cited by
- 114
- References
- 55
- Citations by year
- 2026: 21 · 2025: 15 · 2024: 16 · 2023: 21 · 2022: 21 · 2021: 19
- FWCI
- 4.93
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Nature Biotechnology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- U.S. Department of Health & Human Services | NIH | NIH Office of the Director (S10OD019960)
- Dr. Charles A. Sims’ gift to LLU Center for Genomics
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- single-cell RNA sequencing, benchmarking, reference samples, breast cancer cells, B cells, batch effect correction, normalization methods, preprocessing pipeline, cell classification, gene detection, platform comparison, cellular heterogeneity
- MeSH terms
- Algorithms, B-Lymphocytes, Breast Neoplasms, Female, Humans, Sequence Analysis, RNA, Benchmarking, Gene Expression Profiling, Cell Line, Tumor, Single-Cell Analysis, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
