CoChat
Sign Up
Cited by 436Statistical Methods in Clinical TrialsBioinformatics and Genomic NetworksGene expression and cancer classification

A practical guide to methods controlling false discoveries in computational biology

Open access
Korthauer, Keegan, Kimes, Patrick K., Duvallet, Claire, Reyes, Alejandro, Subramanian, Ayshwarya, Teng, Mingxiang, Shukla, Chinmay, Alm, Eric J. +1 more·2019·Genome Biology
DOI 10.1186/s13059-019-1716-1PMC PMC6547503

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: In high-throughput studies, hundreds to millions of hypotheses are typically tested. Statistical methods that control the false discovery rate (FDR) have emerged as popular and powerful tools for error rate control. While classic FDR methods use only p values as input, more modern FDR methods have been shown to increase power by incorporating complementary information as informative covariates to prioritize, weight, and group hypotheses. However, there is currently no consensus on how the modern methods compare to one another. We investigate the accuracy, applicability, and ease of use of two classic and six modern FDR-controlling methods by performing a systematic benchmark comparison using simulation studies as well as six case studies in computational biology. RESULTS: Methods that incorporate informative covariates are modestly more powerful than classic approaches, and do not underperform classic approaches, even when the covariate is completely uninformative. The majority of methods are successful at controlling the FDR, with the exception of two modern methods under certain settings. Furthermore, we find that the improvement of the modern FDR methods over the classic methods increases with the informativeness of the covariate, total number of hypothesis tests, and proportion of truly non-null hypotheses. CONCLUSIONS: Modern FDR methods that use an informative covariate provide advantages over classic FDR-controlling procedures, with the relative gain dependent on the application and informativeness of available covariates. We present our findings as a practical guide and provide recommendations to aid researchers in their choice of methods to correct for false discoveries.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13059-019-1716-1

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • FDR correction is appropriate and robust for biological data that inherently show dependencies.

    “FDR correction was chosen as it has been shown to be appropriate and robust for biological data that inherently show dependencies, such as those in the present study [60, 61, 62].”

    From Interpersonal Synchronization of Autonomic Physiology via Mere Visual Contact (2026) · cites this paper as [61]

    The Background (page not available) says modern FDR-controlling methods are particularly well suited for biological studies, supporting the appropriateness part. It does not establish that FDR correction is robust to dependencies among biological tests; the dependence discussed in the Results > Power concerns relationships between covariates and the null proportion, and the paper notes that methods have differing assumptions and performance. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. [2]J GJ Aldo S Multiple hypothesis testing in genomics Stat Med 2014 33 11 1946 78 24399688
  2. [3]Genovese CR Lazar NA Nichols T Thresholding of statistical maps in functional neuroimaging using the false discovery rate NeuroImage 2002 15 4 870 878 11906227
  3. [4]Choi H Nesvizhskii AI False discovery rates and related statistical concepts in mass spectrometry-based proteomics J Proteome Res 2007 7 01 47 50 18067251
  4. [6]Keselman H Cribbie R Holland B Controlling the rate of type I error over a large set of statistical tests Br J Math Stat Psychol 2002 55 1 27 39 12034010
  5. [15]Ignatiadis N Klaus B Zaugg JB Huber W Data-driven hypothesis weighting increases detection power in genome-scale multiple testing Nat Methods 2016 13 577 80 27240256 PMC4930141
  6. [16]Boca SM, Leek JT. A direct approach to estimating false discovery rates conditional on covariates. bioRxiv. 2017. Available from: 10.1101/035675. PMC6292380 30581661 · DOI
  7. [19]Scott JG Kelly RC Smith MA Zhou P Kass RE False discovery rate regression: an application to neural synchrony detection in primary visual cortex J Am Stat Assoc 2015 110 459 71 26855459 PMC4743052
  8. [20]Stephens M False discovery rates: a new deal Biostatistics 2016 18 275 94 PMC5379932 27756721
  9. [22]Chen JJ Robeson PK Schell MJ The false discovery rate: a key concept in large-scale genetic studies Canc Control 2010 17 1 58 62 20010520
  10. [27]Chen X, Robinson DG, Storey JD. The functional false discovery rate with applications to genomics. bioRxiv. 2017. Available from: 10.1101/241133. PMC7846131 31135886 · DOI
  11. [28]Frazee AC Jaffe AE Langmead B Leek JT Polyester: simulating RNA-seq datasets with differential transcript expression Bioinformatics 2015 31 17 2778 84 25926345 PMC4635655
  12. [29]Gierliński M Cole C Schofield P Schurch NJ Sherstnev A Singh V Statistical models for RNA-seq data derived from a two-condition 48-replicate experiment Bioinformatics 2015 31 22 3625 30 26206307 PMC4754627
  13. [31]Soneson C Robinson MD iCOBRA: open, reproducible, standardized and live method benchmarking Nat Methods 2016 13 4 283 27027585
  14. [32]Ritchie ME Phipson B Wu D Hu Y Law CW Shi W limma powers differential expression analyses for RNA-sequencing and microarray studies Nucleic Acids Res 2015 43 7 e47 7 25605792 PMC4402510
  15. [33]Lu M, Stephens M. Empirical Bayes estimation of normal means, accounting for uncertainty in estimated standard errors. arXiv. 2019. Available from: https://arxiv.org/1901.10679.
  16. [34]Bourgon R Gentleman R Huber W Independent filtering increases detection power for high-throughput experiments Proc Nat Acad Sci 2010 107 21 9546 51 20460310 PMC2906865
  17. [36]Kimes PK Reyes A Reproducible and replicable comparisons using SummarizedBenchmark Bioinformatics 2018 35 1 137 39 PMC6298041 30016409
  18. [38]Li A, Barber RF. Multiple testing with the structure adaptive Benjamini-Hochberg algorithm. arXiv. 2017. Available from: https://arxiv.org/1606.07926.
  19. [40]Speliotes EK Willer CJ Berndt KL S I Monda Thorleifsson G Jackson AU Allen CM H L Lindgren Association analyses of 249,796 individuals reveal 18 new loci associated with body mass index Nat Gene 2010 42 11 937 48 PMC3014648 20935630
  20. [42]Deloukas P Kanoni S Willenborg C Farrall M Assimes TL Thompson JR Large-scale association analysis identifies new risk loci for coronary artery disease Nat Gene 2013 45 1 25 33 PMC3679547 23202125
  21. [45]Mele M. Ferreira P. G. Reverter F. DeLuca D. S. Monlong J. Sammeth M. Young T. R. Goldmann J. M. Pervouchine D. D. Sullivan T. J. Johnson R. Segre A. V. Djebali S. Niarchou A. Consortium T. G. Wright F. A. Lappalainen T. Calvo M. Getz G. Dermitzakis E. T. Ardlie K. G. Guigo R. The human transcriptome across tissues and individuals Science 2015 348 6235 660 665 25954002 PMC4547472
  22. [46]Cabezas-Wallscheid Nina Klimmeck Daniel Hansson Jenny Lipka Daniel B. Reyes Alejandro Wang Qi Weichenhan Dieter Lier Amelie von Paleske Lisa Renders Simon Wünsche Peer Zeisberger Petra Brocks David Gu Lei Herrmann Carl Haas Simon Essers Marieke A.G. Brors Benedikt Eils Roland Huber Wolfgang Milsom Michael D. Plass Christoph Krijgsveld Jeroen Trumpp Andreas Identification of Regulatory Networks in HSCs and Their Immediate Progeny via Integrated Proteome, Transcriptome, and DNA Methylome Analysis Cell Stem Cell 2014 15 4 507 522 25158935
  23. [50]Sergushichev A. An algorithm for fast preranked gene set enrichment analysis using cumulative statistic calculation. BioRxiv. 2016. Available from: 10.1101/060012. · DOI
  24. [51]Reyes Alejandro Huber Wolfgang Alternative start and termination sites of transcription drive most transcript isoform differences across human tissues Nucleic Acids Research 2017 46 2 582 592 PMC5778607 29202200
  25. [52]Kim YK, Wee G, Park J, Kim J, Baek D, Kim JS, et al. TALEN-based knockout library for human microRNAs. Nat Struct & Mole Biol. 2013; 20(12):1458–64. Available from: 10.1038/nsmb.2701. 24213537 · DOI
  26. [53]Collado-Torres Leonardo Nellore Abhinav Kammers Kai Ellis Shannon E Taub Margaret A Hansen Kasper D Jaffe Andrew E Langmead Ben Leek Jeffrey T Reproducible RNA-seq analysis using recount2 Nature Biotechnology 2017 35 4 319 321 28398307 PMC6742427
  27. [55]Soneson C Robinson MD Bias, robustness and scalability in single-cell differential expression analysis Nat Methods 2018 15 4 255 61 29481549
  28. [57]Darmanis S Sloan SA Croote D Mignardi M Chernikova S Samghababi P Single-cell RNA-Seq analysis of infiltrating neoplastic cells at the migrating front of human glioblastoma Cell Rep 2017 21 5 1399 410 29091775 PMC5810554
  29. [58]Lane K Van Valen D DeFelice MM Macklin DN Kudo T Jaimovich A Measuring signaling and RNA-seq in the same cell links gene expression to dynamic patterns of NF- κB activation Cell Syst 2017 4 4 458 69 28396000 PMC6748049
  30. [60]Finak G McDavid A Yajima M Deng J Gersuk V Shalek AK MAST: a flexible statistical framework for assessing transcriptional changes and characterizing heterogeneity in single-cell RNA sequencing data Genome Biol 2015 16 1 278 26653891 PMC4676162
  31. [61]Lun AT McCarthy DJ Marioni JC A step-by-step workflow for low-level analysis of single-cell RNA-seq data with Bioconductor F1000Research 2016 5 2122 27909575 PMC5112579
  32. [62]Robinson M. D. McCarthy D. J. Smyth G. K. edgeR: a Bioconductor package for differential expression analysis of digital gene expression data Bioinformatics 2009 26 1 139 140 19910308 PMC2796818
  33. [63]ENCODE Project Consortium An integrated encyclopedia of DNA elements in the human genome Nature 2012 489 7414 57 22955616 PMC3439153
  34. [65]Lun AT Smyth GK From reads to regions: a bioconductor workflow to detect differential binding in ChIP-seq data F1000Research 2015 4 1080 26834993 PMC4706055
  35. [66]Lun AT Smyth GK csaw: a Bioconductor package for differential binding analysis of ChIP-seq data using sliding windows Nucleic Acids Res 2015 44 5 e45 e45 26578583 PMC4797262
  36. [67]Kasper LH Qu C Obenauer JC McGoldrick DJ Brindle PK Genome-wide and single-cell analyses reveal a context dependent relationship between CBP recruitment and gene expression Nucleic Acids Res 2014 42 18 11363 82 25249627 PMC4191404
  37. [69]Duvallet C Gibbons SM Gurry T Irizarry RA Alm EJ Meta-analysis of gut microbiome studies identifies disease-specific and shared responses Nat Commun 2017 8 1 1784 29209090 PMC5716994
  38. [70]Duvallet C, Gibbons S, Gurry T, Irizarry R, Alm E; 2017. 10.5281/zenodo.840333. PMC5716994 29209090 · DOI
  39. [72]Papa E Docktor M Smillie C Weber S Preheim SP Gevers D Non-invasive mapping of the gastrointestinal microbiota identifies children with inflammatory bowel disease PLoS ONE 2012 7 6 e39242 22768065 PMC3387146
  40. [73]Schubert AM Rogers MAM Ring C Mogle J Petrosino JP Young VB Microbiome data distinguish patients with Clostridium difficile infection and non-C. difficile-associated diarrhea from healthy controls mBio 2014 5 3 e01021 14 24803517 PMC4010826
  41. [74]Baxter NT, Ruffin MT, Rogers MAM, Schloss PD. Microbiota-based model improves the sensitivity of fecal immunochemical test for detecting colonic lesions. Genome Med. 2016;8(1). PMC4823848 27056827
  42. [75]Smith MB, Rocha AM, Smillie CS, Olesen SW, Paradis C, Wu L, et al. Natural bacterial communities serve as quantitative geochemical biosensors. mBio. 2015; 6(3):e00326–15. PMC4436078 25968645
  43. [76]Zhang J Kobert K Flouri T Stamatakis A PEAR: a fast and accurate Illumina Paired-End reAd mergeR Bioinformatics 2013 30 5 614 20 24142950 PMC3933873
  44. [77]Caporaso JG Kuczynski J Stombaugh J Bittinger K Bushman FD Costello EK QIIME allows analysis of high-throughput community sequencing data Nat Methods 2010 7 5 335 20383131 PMC3156573
  45. [78]Edgar RC Search and clustering orders of magnitude faster than BLAST Bioinformatics 2010 26 19 2460 61 20709691
  46. [81]Korthauer K, Kimes PK, Duvallet C, Reyes A, Subramanian A, Teng M, et al. Shiny app for exploring results from “A practical guide to methods controlling false discoveries in computational biology”: GitHub; 2019. https://github.com/kdkorthauer/benchmarkfdr-shiny/commit/4ce60ed1a6b36e681b63b6c244317bc40de39ccd. Accessed 22 May 2019. PMC6547503 31164141

Cited by

Sorted by citations

Authors
Korthauer, K., Kimes, P. K., Duvallet, C., Reyes, A., Subramanian, A., Teng, M., Shukla, C., Alm, E. J., & Hicks, S. C.
Affiliations
  • Harvard University (US)
  • Massachusetts Institute of Technology (US)
  • Broad Institute (US)
  • Johns Hopkins University (US)
Corresponding author
Hicks, Stephanie C.
ORCID
Venue
Genome Biology
Publisher
Springer Science and Business Media LLC
Published
2019-06-04
Volume
20
Issue
1
Pages
118-118
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
DOI
10.1186/s13059-019-1716-1
PMC
PMC6547503
OpenAlex
W2951144892
Cited by
436
References
8
Citations by year
2026: 49 · 2025: 84 · 2024: 92 · 2023: 73 · 2022: 61 · 2021: 52 · 2020: 12 · 2019: 10 · 2018: 2
FWCI
27.50
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Funders
  • National Human Genome Research Institute (U41HG004059, R02HG005220, R00HG009007)
  • National Institute of General Medical Sciences (R01GM083084)
  • National Institue of General Medical Sciences (R01GM103552)
  • Chan Zuckerberg Initiative DAF (2018-183142, 2018-183560)
  • Broadnext10 (Broadnext10)
  • National Cancer Institute (P30CA076292)
  • Office of Science (DE-AC02-05CH11231)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
Keywords
false discovery rate, multiple comparisons, statistical power, simulation study, computational biology
MeSH terms
Computer Simulation, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review