Exaggerated false positives by popular differential expression methods when analyzing human population samples
Open accessPublished version of a preprint
Abstract
When identifying differentially expressed genes between two conditions using human population RNA-seq samples, we found a phenomenon by permutation analysis: two popular bioinformatics methods, DESeq2 and edgeR, have unexpectedly high false discovery rates. Expanding the analysis to limma-voom, NOISeq, dearseq, and Wilcoxon rank-sum test, we found that FDR control is often failed except for the Wilcoxon rank-sum test. Particularly, the actual FDRs of DESeq2 and edgeR sometimes exceed 20% when the target FDR is 5%. Based on these results, for population-level RNA-seq studies with large sample sizes, we recommend the Wilcoxon rank-sum test.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Parametric models for gene expression distribution can yield miscalibrated p-values when misspecified.
“methods that rely upon parametric models for the gene expression distribution, such as NB regression and MAST, can yield miscalibrated p-values when those models are misspecified [27]”
From Robust differential expression testing for single-cell CRISPR screens at low multiplicity of infection (2024) · cites this paper as [27]
The Results and discussion section states that violating parametric assumptions can produce ill-behaved p-values, and links poor negative-binomial model fit to spurious discoveries by DESeq2 and edgeR. This supports the general claim and the negative-binomial example, but our copy does not address MAST, which the citing sentence also names. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2
Love, M. I., Huber, W., & Anders, S. (2014) · Genome Biology · doi:10.1186/s13059-014-0550-8 · Cited by 95,106 · see citing works
Robinson, M. D., McCarthy, D. J., & Smyth, G. K. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btp616 · Cited by 41,364 · see citing works
limma powers differential expression analyses for RNA-sequencing and microarray studies
Ritchie, M. E., Phipson, B., Wu, D., Hu, Y., Law, C. W., Shi, W., & Smyth, G. K. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv007 · Cited by 40,458 · see citing works
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320 · see citing works
RNA-Seq: a revolutionary tool for transcriptomics
Wang, Z., Gerstein, M., & Snyder, M. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2484 · Cited by 13,174 · see citing works
GEPIA: a web server for cancer and normal gene expression profiling and interactive analyses
Tang, Z., Li, C., Kang, B., Gao, G., Li, C., & Zhang, Z. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx247 · Cited by 9,849 · see citing works
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092 · see citing works
voom: precision weights unlock linear model analysis tools for RNA-seq read counts
Law, C. W., Chen, Y., Shi, W., & Smyth, G. K. (2014) · Genome Biology · doi:10.1186/gb-2014-15-2-r29 · Cited by 6,155 · see citing works
The GTEx Consortium atlas of genetic regulatory effects across human tissues
The GTEx Consortium, Aguet, F., Anand, S., Ardlie, K. G., Gabriel, S., Getz, G. A., Graubert, A., Hadley, K., Handsaker, R. E., Huang, K. H., et al. (2020) · Science · doi:10.1126/science.aaz1776 · Cited by 5,547 · see citing works
Self-citationThe Transcriptional Landscape of the Yeast Genome Defined by RNA Sequencing
Nagalakshmi, U., Wang, Z., Waern, K., Shou, C., Raha, D., Gerstein, M., & Snyder, M. (2008) · Science · doi:10.1126/science.1158441 · Cited by 2,479 · see citing works
Tumor and Microenvironment Evolution during Immunotherapy with Nivolumab
Riaz, N., Havel, J. J., Makarov, V., Desrichard, A., Urba, W. J., Sims, J. S., Hodi, F. S., Martín-Algarra, S., Mandal, R., Sharfman, W. H., et al. (2017) · Cell · doi:10.1016/j.cell.2017.09.028 · Cited by 2,401 · see citing works
RNA sequencing: the teenage years
Stark, R., Grzelak, M., & Hadfield, J. (2019) · Nature Reviews Genetics · doi:10.1038/s41576-019-0150-2 · Cited by 1,906 · see citing works
Bullard, J. H., Purdom, E., Hansen, K. D., & Dudoit, S. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-94 · Cited by 1,680 · see citing works
The human transcriptome across tissues and individuals
Melé, M., Ferreira, P. G., Reverter, F., DeLuca, D. S., Monlong, J., Sammeth, M., Young, T. R., Goldmann, J. M., Pervouchine, D. D., Sullivan, T. J., et al. (2015) · Science · doi:10.1126/science.aaa0355 · Cited by 1,304 · see citing works
Fay, M. P., & Proschan, M. A. (2010) · Statistics Surveys · doi:10.1214/09-ss051 · Cited by 1,121 · see citing works
Gide, T. N., Quek, C., Menzies, A. M., Tasker, A. T., Shang, P., Holst, J., Madore, J., Lim, S. Y., Velickovic, R., Wongchenko, M., et al. (2019) · Cancer Cell · doi:10.1016/j.ccell.2019.01.003 · Cited by 1,047 · see citing works
Schurch, N. J., Schofield, P., Gierliński, M., Cole, C., Sherstnev, A., Singh, V., Wrobel, N., Gharbi, K., Simpson, G. G., Owen-Hughes, T., et al. (2016) · RNA · doi:10.1261/rna.053959.115 · Cited by 966 · see citing works
A comparison of methods for differential expression analysis of RNA-seq data
Soneson, C., & Delorenzi, M. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-91 · Cited by 887 · see citing works
Data quality aware analysis of differential expression in RNA-seq with NOISeq R/Bioc package
Tarazona, S., Furió‐Tarí, P., Turrà, D., Pietro, A. D., Nueda, M. J., Ferrer, A., & Conesa, A. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv711 · Cited by 757 · see citing works
Comprehensive evaluation of differential gene expression analysis methods for RNA-seq data
Rapaport, F., Khanin, R., Liang, Y., Pirun, M., Krek, A., Zumbo, P., Mason, C. E., Socci, N. D., & Betel, D. (2013) · Genome Biology · doi:10.1186/gb-2013-14-9-r95 · Cited by 733 · see citing works
RNA-Seq differential expression analysis: An extended review and a software tool
Silva, J. C., Domingues, D. S., & Lopes, F. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190152 · Cited by 624 · see citing works
The Efficiency of Some Nonparametric Competitors of the $t$-Test
Hodges, J. L., & Lehmann, E. L. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728261 · Cited by 589 · see citing works
t-tests, non-parametric tests, and large studies—a paradox of statistical practice?
Fagerland, M. W. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-78 · Cited by 549 · see citing works
Asymptotic Normality and Efficiency of Certain Nonparametric Test Statistics
Chernoff, H., & Savage, I. R. (1958) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706436 · Cited by 513 · see citing works
Individual Comparisons of Grouped Data by Ranking Methods
Wilcoxon, F. (1946) · Journal of Economic Entomology · doi:10.1093/jee/39.2.269 · Cited by 482 · see citing works
Comparison of software packages for detecting differential expression in RNA-seq studies
Seyednasrollah, F., Laiho, A., & Elo, L. L. (2015) · Briefings in Bioinformatics · doi:10.1093/bib/bbt086 · Cited by 421 · see citing works
A Comparative Study of Techniques for Differential Expression Analysis on RNA-Seq Data
Zhang, Z. H., Jhaveri, D. J., Marshall, V. M., Bauer, D. C., Edson, J., Narayanan, R. K., Robinson, G. J., Lundberg, A. E., Bartlett, P. F., Wray, N. R., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103207 · Cited by 357 · see citing works
A comparison of statistical methods for detecting differentially expressed genes from RNA‐seq data
Kvam, V. M., Liu, P., & Si, Y. (2012) · American Journal of Botany · doi:10.3732/ajb.1100340 · Cited by 281 · see citing works
Pan-cancer analysis of transcriptional metabolic dysregulation using The Cancer Genome Atlas
Rosario, S. R., Long, M. D., Affronti, H. C., Rowsam, A. M., Eng, K. H., & Smiraglia, D. J. (2018) · Nature Communications · doi:10.1038/s41467-018-07232-8 · Cited by 268 · see citing works
Systematic comparison and assessment of RNA-seq procedures for gene expression quantitative analysis
Corchete, L. A., Rojas, E. A., Alonso‐López, D., Rivas, J. D. L., Gutierrez, N. C., & Burguillo, F. J. (2020) · Scientific Reports · doi:10.1038/s41598-020-76881-x · Cited by 250 · see citing works
Thas, O., Neve, J. D., Clement, L., & Ottoy, J. (2012) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2011.01020.x · Cited by 160 · see citing works
Peng, L. J., Bian, X. W., Li, D. K., Xu, C. S., Wang, G., Xia, Q. Y., & Xiong, Q. (2015) · Scientific Reports · doi:10.1038/srep13413 · Cited by 110 · see citing works
Quinn, T. P., Crowley, T. M., & Richardson, M. F. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2261-8 · Cited by 92 · see citing works
Williams, C., Baccarella, A., Parrish, J. Z., & Kim, C. C. (2017) · BMC Bioinformatics · doi:10.1186/s12859-016-1457-z · Cited by 83 · see citing works
Li, X., Cooper, N. G. F., O’Toole, T. E., & Rouchka, E. C. (2020) · BMC Genomics · doi:10.1186/s12864-020-6502-7 · Cited by 55 · see citing works
Clipper: p-value-free FDR control on high-throughput data from two conditions
Ge, X., Chen, Y. E., Song, D., McDermott, M., Woyshner, K., Manousopoulou, A., Wang, N., Li, W., Wang, L. D., & Li, J. J. (2021) · Genome biology · doi:10.1186/s13059-021-02506-9 · Cited by 52 · see citing works
Self-citationBenchmarking RNA-seq differential expression analysis methods using spike-in and simulation data
Baik, B., Yoon, S., & Nam, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0232271 · Cited by 47 · see citing works
Sequence count data are poorly fit by the negative binomial distribution
Hawinkel, S., Rayner, J. C. W., Bijnens, L., & Thas, O. (2020) · PLoS ONE · doi:10.1371/journal.pone.0224909 · Cited by 40 · see citing works
Inflated false discovery rate due to volcano plots: problem and solutions
Ebrahimpoor, M., & Goeman, J. (2021) · Briefings Bioinform. · doi:10.1093/bib/bbab053 · Cited by 34 · see citing works
Gauthier, M., Agniel, D. M., Thiébaut, R., & Hejblum, B. P. (2020) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqaa093 · Cited by 33 · see citing works
An extension of the Wilcoxon-Mann-Whitney test for analyzing RT-qPCR data
Neve, J. D., Thas, O., Ottoy, J., et al. (2013) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/sagmb-2012-0003 · Cited by 19 · see citing works
Processed datasets and codes for differential expression analysis on polulation-level RNA-seq data
Li, Y., & Ge, X. (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5241320 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [45]Li Y, Ge X. Exaggerated false positives by popular differential expression methods when analyzing human population samples. Github. 2022; https://github.com/xihuimeijing/DEGs_Analysis_FDR. PMC8922736 35292087
Cited by
Sorted by citations
Barry, T., Mason, K., Roeder, K., & Katsevich, E. (2024) · Genome Biology · doi:10.1186/s13059-024-03254-2 · Cited by 28 · see citing works
Burger, T. (2023) · PROTEOMICS · doi:10.1002/pmic.202200406 · Cited by 10 · see citing works
“excessive trust in moderated statistics could be damaging [7], and guidance about when and how”
Background
- Authors
- Li, Y., Ge, X., Peng, F., Li, W., & Li, J. J.
- Affiliations
- University of California, Irvine (US)
- University of California, Los Angeles (US)
- Baylor College of Medicine (US)
- Corresponding author
- Li, Wei
- Li, Jingyi Jessica
- Venue
- Genome Biology
- Published
- 2022-03-15
- Volume
- 23
- Issue
- 1
- Pages
- 79-79
- ISSN
- 1465-6906 (ISSN-L)
- 1474-760X
- PMC
- PMC8922736
- OpenAlex
- W4221068576
- Cited by
- 303
- References
- 43
- Citations by year
- 2026: 48 · 2025: 93 · 2024: 83 · 2023: 60 · 2022: 18 · 2021: 1
- FWCI
- 21.04
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Genome biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- National Cancer Institute (R01CA193466, R01CA228140)
- National Institute of General Medical Sciences (R01GM120507, R35GM140888)
- Division of Biological Infrastructure (1846216)
- Division of Mathematical Sciences (2113754)
- Johnson and Johnson (WiSTEM2D Award)
- Alfred P. Sloan Foundation (Sloan Research Fellowship)
- W. M. Keck Foundation (UCLA David Geffen School of Medicine W.M. Keck Foundation Junior Faculty Award)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- false discovery rate, differential gene expression, DESeq2, edgeR, Mann-Whitney test, permutation analysis, false positives, false discovery rate control
- MeSH terms
- RNA-Seq, Humans, Sequence Analysis, RNA, Sample Size, Computational Biology, Gene Expression Profiling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
