On testing the significance of sets of genes
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
This paper discusses the problem of identifying differentially expressed groups of genes from a microarray experiment. The groups of genes are externally defined, for example, sets of gene pathways derived from biological databases. Our starting point is the interesting Gene Set Enrichment Analysis (GSEA) procedure of Subramanian et al. [Proc. Natl. Acad. Sci. USA 102 (2005) 15545–15550]. We study the problem in some generality and propose two potential improvements to GSEA: the maxmean statistic for summarizing gene-sets, and restandardization for more accurate inferences. We discuss a variety of examples and extensions, including the use of gene-set scores for class predictions. We also describe a new R language package GSA that implements our ideas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
Smyth, G. K. (2004) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1027 · Cited by 11,829
In silico prediction of protein-protein interactions in human macrophages
Souiai, O., Guerfali, F., Ben Miled, S., Brun, C., & Benkahla, A. (2014) · BMC Research Notes · doi:10.1186/1756-0500-7-157 · Cited by 10,803
Significance analysis of microarrays applied to the ionizing radiation response
Tusher, V. G., Tibshirani, R., & Chu, G. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.091062498 · Cited by 10,568
Self-citationDiagnosis of multiple cancer types by shrunken centroids of gene expression
Tibshirani, R., Hastie, T., Narasimhan, B., & Chu, G. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.082099299 · Cited by 2,930
Self-citationGene-Expression Profiles in Hereditary Breast Cancer
Hedenfalk, I., Duggan, D., Chen, Y., Radmacher, M., Bittner, M., Simon, R., Meltzer, P., Gusterson, B., Esteller, M., Raffeld, M., et al. (2001) · New England Journal of Medicine · doi:10.1056/nejm200102223440801 · Cited by 1,733
Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesis
Efron, B. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000000089 · Cited by 1,005
Self-citationDiscovering statistically significant pathways in expression profiling studies
Tian, L., Greenberg, S. A., Kong, S. W., Altschuler, J., Kohane, I. S., & Park, P. J. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506577102 · Cited by 648
Barry, W. T., Nobel, A. B., & Wright, F. A. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti260 · Cited by 356
Cited by
GSVA: gene set variation analysis for microarray and RNA-Seq data
Hänzelmann, S., Castelo, R., & Guinney, J. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-7 · Cited by 15,440
The consensus molecular subtypes of colorectal cancer
Guinney, J., Dienstmann, R., Wang, X., de Reyniès, A., Schlicker, A., Soneson, C., Marisa, L., Roepman, P., Nyamundanda, G., Angelino, P., et al. (2015) · Nature Medicine · doi:10.1038/nm.3967 · Cited by 5,170
“is supported by grants from the Dutch Cancer Society (UVA2011-4969, UVA2014-7245), Worldwide Cancer Research (14-1164), the Maag Lever Darm Stichting (MLDS-CDG 14-03) and the European Research Council (ERG-StG 638193).”
Radiation and dual checkpoint blockade activate non-redundant immune mechanisms in cancer
Twyman-Saint Victor, C., Rech, A. J., Maity, A., Rengan, R., Pauken, K. E., Stelekati, E., Benci, J. L., Xu, B., Dada, H., Odorizzi, P. M., et al. (2015) · Nature · doi:10.1038/nature14292 · Cited by 2,434
“To test whether gene sets were enriched in response to different conditions, we utilized Gene Set Analysis as implemented in the “GSA” R package version 1.03 29 .”
InfluentialUsing MetaboAnalyst 4.0 for Comprehensive and Integrative Metabolomics Data Analysis
Chong, J., Wishart, D. S., & Xia, J. (2019) · Current Protocols in Bioinformatics · doi:10.1002/cpbi.86 · Cited by 2,166
Bu, D., Luo, H., Huo, P., Wang, Z., Zhang, S., He, Z., Wu, Y., Zhao, L., Liu, J., Guo, J., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab447 · Cited by 1,885
Ten Years of Pathway Analysis: Current Approaches and Outstanding Challenges
Khatri, P., Sirota, M., & Butte, A. J. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002375 · Cited by 1,566
Using MetaboAnalyst 3.0 for Comprehensive Metabolomics Data Analysis
Xia, J., & Wishart, D. S. (2016) · Current Protocols in Bioinformatics · doi:10.1002/cpbi.11 · Cited by 1,544
Deletion of IKZF1 and Prognosis in Acute Lymphoblastic Leukemia
Mullighan, C. G., Su, X., Zhang, J., Radtke, I., Phillips, L. A. A., Miller, C. B., Ma, J., Liu, W., Cheng, C., Schulman, B. A., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0808253 · Cited by 1,422
Whole-Genome and Epigenomic Landscapes of Etiologically Distinct Subtypes of Cholangiocarcinoma
Jusakul, A., Cutcutache, I., Yong, C. H., Lim, J. Q., Huang, M. N., Padmanabhan, N., Nellore, V., Kongpetch, S., Ng, A. W. T., Ng, L. M., et al. (2017) · Cancer Discovery · doi:10.1158/2159-8290.cd-17-0368 · Cited by 969
Camera: a competitive gene set test accounting for inter-gene correlation
Wu, D., & Smyth, G. K. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks461 · Cited by 891
Exosome Transfer from Stromal to Breast Cancer Cells Regulates Therapy Resistance Pathways
Boelens, M. C., Wu, T. J., Nabet, B. Y., Xu, B., Qiu, Y., Yoon, T., Azzam, D. J., Twyman-Saint Victor, C., Wiemann, B. Z., Ishwaran, H., et al. (2014) · Cell · doi:10.1016/j.cell.2014.09.051 · Cited by 806
MSEA: a web-based tool to identify biologically meaningful patterns in quantitative metabolomic data
Xia, J., & Wishart, D. S. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq329 · Cited by 782
Analysing biological pathways in genome-wide association studies
Wang, K., Li, M., & Hakonarson, H. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2884 · Cited by 766
Feature Screening via Distance Correlation Learning
Li, R., Zhong, W., & Zhu, L. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.695654 · Cited by 747
Väremo, L., Nielsen, J., & Nookaew, I. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt111 · Cited by 697
Detection of gene pathways with predictive power for breast cancer prognosis
Ma, S., & Kosorok, M. R. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-1 · Cited by 690
A two-sample test for high-dimensional data with applications to gene-set testing
Chen, S. X., & Qin, Y.-L. (2010) · The Annals of Statistics · doi:10.1214/09-aos716 · Cited by 574
Identification of senescent cells in the bone microenvironment
Farr, J. N., Fraser, D. G., Wang, H., Jaehn, K., Ogrodnik, M. B., Weivoda, M. M., Drake, M. T., Tchkonia, T., LeBrasseur, N. K., Kirkland, J. L., et al. (2016) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2892 · Cited by 562
Johnson, D. B., Estrada, M. V., Salgado, R., Sanchez, V., Doxie, D. B., Opalenik, S. R., Vilgelm, A. E., Feld, E., Johnson, A. S., Greenplate, A. R., et al. (2016) · Nature Communications · doi:10.1038/ncomms10582 · Cited by 566
ROAST: rotation gene set tests for complex microarray experiments
Wu, D., Lim, E., Vaillant, F., Asselin-Labat, M.-L., Visvader, J. E., & Smyth, G. K. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq401 · Cited by 560
CellNet: Network Biology Applied to Stem Cell Engineering
Cahan, P., Li, H., Morris, S. A., Lummertz da Rocha, E., Daley, G. Q., & Collins, J. J. (2014) · Cell · doi:10.1016/j.cell.2014.07.020 · Cited by 530
Plaisier, S. B., Taschereau, R., Wong, J. A., & Graeber, T. G. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq636 · Cited by 468
Chemotherapy-induced antitumor immunity requires formyl peptide receptor 1
Vacchelli, E., Ma, Y., Baracco, E. E., Sistigu, A., Enot, D. P., Pietrocola, F., Yang, H., Adjemian, S., Chaba, K., Semeraro, M., et al. (2015) · Science · doi:10.1126/science.aad0779 · Cited by 452
Danaher, P., Warren, S., Lu, R., Samayoa, J., Sullivan, A., Pekker, I., Wallden, B., Marincola, F. M., & Cesano, A. (2018) · Journal for ImmunoTherapy of Cancer · doi:10.1186/s40425-018-0367-1 · Cited by 421
Data integration in the era of omics: current and future challenges
Gomez-Cabrero, D., Abugessaisa, I., Maier, D., Teschendorff, A., Merkenschlager, M., Gisel, A., Ballestar, E., Bongcam-Rudloff, E., Conesa, A., & Tegnér, J. (2014) · BMC Systems Biology · doi:10.1186/1752-0509-8-s2-i1 · Cited by 413
Effective breast cancer combination therapy targeting BACH1 and mitochondrial metabolism
Lee, J., Yesilkanal, A. E., Wynne, J. P., Frankenberger, C., Liu, J., Yan, J., Elbaz, M., Rabe, D. C., Rustandy, F. D., Tiwari, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1005-x · Cited by 380
A general modular framework for gene set enrichment analysis
Ackermann, M., & Strimmer, K. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-47 · Cited by 358
Regulatory networks define phenotypic classes of human stem cell lines
Müller, F.-J., Laurent, L. C., Kostka, D., Ulitsky, I., Williams, R., Lu, C., Park, I.-H., Rao, M. S., Shamir, R., Schwartz, P. H., et al. (2008) · Nature · doi:10.1038/nature07213 · Cited by 343
Cellular Senescence Is Immunogenic and Promotes Antitumor Immunity
Marin, I., Boix, O., Garcia-Garijo, A., Sirois, I., Caballe, A., Zarzuela, E., Ruano, I., Attolini, C. S.-O., Prats, N., López-Domínguez, J. A., et al. (2023) · Cancer Discovery · doi:10.1158/2159-8290.cd-22-0523 · Cited by 346
Jalanka-Tuovinen, J., Salojärvi, J., Salonen, A., Immonen, O., Garsed, K., Kelly, F. M., Zaitoun, A., Palva, A., Spiller, R. C., & de Vos, W. M. (2014) · Gut · doi:10.1136/gutjnl-2013-305994 · Cited by 340
Gene-set approach for expression pattern analysis
Nam, D., & Kim, S.-Y. (2008) · Briefings in Bioinformatics · doi:10.1093/bib/bbn001 · Cited by 338
Mavrakis, K. J., Wolfe, A. L., Oricchio, E., Palomero, T., de Keersmaecker, K., McJunkin, K., Zuber, J., James, T., Khan, A. A., Leslie, C. S., et al. (2010) · Nature Cell Biology · doi:10.1038/ncb2037 · Cited by 337
Tests for High-Dimensional Covariance Matrices
Chen, S. X., Zhang, L.-X., & Zhong, P.-S. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2010.tm09560 · Cited by 329
Integrated genomic characterization of IDH1-mutant glioma malignant progression
Bai, H., Harmancı, A. S., Erson-Omay, E. Z., Li, J., Coşkun, S., Simon, M., Krischek, B., Özduman, K., Omay, S. B., Sorensen, E. A., et al. (2016) · Nature Genetics · doi:10.1038/ng.3457 · Cited by 329
Herazo-Maya, J. D., Noth, I., Duncan, S. R., Kim, S., Ma, S.-F., Tseng, G. C., Feingold, E., Juan-Guardela, B. M., Richards, T. J., Lussier, Y., et al. (2013) · Science Translational Medicine · doi:10.1126/scitranslmed.3005964 · Cited by 325
Molecular Classification of Gastric Cancer: A New Paradigm
Shah, M. A., Khanin, R., Tang, L., Janjigian, Y. Y., Klimstra, D. S., Gerdes, H., & Kelsen, D. P. (2011) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-10-2203 · Cited by 311
Stable feature selection for biomarker discovery
He, Z., & Yu, W. (2010) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2010.07.002 · Cited by 309
Dietary palmitic acid promotes a prometastatic memory via Schwann cells
Pascual, G., Domínguez, D., Elosúa-Bayes, M., Beckedorff, F., Laudanna, C., Bigas, C., Douillet, D., Greco, C., Symeonidi, A., Hernández, I., et al. (2021) · Nature · doi:10.1038/s41586-021-04075-0 · Cited by 308
Metabolomic Data Processing, Analysis, and Interpretation Using MetaboAnalyst
Xia, J., & Wishart, D. S. (2011) · Current Protocols in Bioinformatics · doi:10.1002/0471250953.bi1410s34 · Cited by 288
Metabolic Phenotypes of Response to Vaccination in Humans
Li, S., Sullivan, N. L., Rouphael, N., Yu, T., Banton, S., Maddur, M. S., McCausland, M., Chiu, C., Canniff, J., Dubey, S., et al. (2017) · Cell · doi:10.1016/j.cell.2017.04.026 · Cited by 276
Keap1 Mutations and Nrf2 Pathway Activation in Epithelial Ovarian Cancer
Konstantinopoulos, P. A., Spentzos, D., Fountzilas, E., Francoeur, N., Sanisetty, S., Grammatikos, A. P., Hecht, J. L., & Cannistra, S. A. (2011) · Cancer Research · doi:10.1158/0008-5472.can-10-4668 · Cited by 270
Núñez, F. J., Mendez, F. M., Kadiyala, P., Alghamri, M. S., Savelieff, M. G., Garcia-Fabiani, M. B., Haase, S., Koschmann, C., Calinescu, A.-A., Kamran, N., et al. (2019) · Science Translational Medicine · doi:10.1126/scitranslmed.aaq1427 · Cited by 269
Farr, J. N., & Khosla, S. (2019) · Bone · doi:10.1016/j.bone.2019.01.015 · Cited by 263
Gene set enrichment analysis: performance evaluation and usage guidelines
Hung, J.-H., Yang, T.-H., Hu, Z., Weng, Z., & DeLisi, C. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbr049 · Cited by 266
Pathway Analysis: State of the Art
García-Campos, M. A., Espinal-Enríquez, J., & Hernández-Lemus, E. (2015) · Frontiers in Physiology · doi:10.3389/fphys.2015.00383 · Cited by 265
Sveen, A., Bruun, J., Eide, P. W., Eilertsen, I. A., Ramirez, L., Murumägi, A., Arjama, M., Danielsen, S. A., Kryeziu, K., Elez, E., et al. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-17-1234 · Cited by 256
“Gene set expression enrichment analyses were performed using the R package GSA (30) and a customized collection of 51 colorectal cancer-related gene sets.”
MethodMetastatic recurrence in colorectal cancer arises from residual EMP1+ cells
Cañellas-Socias, A., Cortina, C., Hernando-Momblona, X., Palomo-Ponce, S., Mulholland, E. J., Turon, G., Mateo, L., Conti, S., Roman, O., Sevillano, M., et al. (2022) · Nature · doi:10.1038/s41586-022-05402-9 · Cited by 256
Microarrays, Empirical Bayes and the Two-Groups Model
Efron, B. (2008) · Statistical Science · doi:10.1214/07-sts236 · Cited by 252
Self-citationEffects of dependence in high-dimensional multiple testing problems
Kim, K. I., & van de Wiel, M. A. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-114 · Cited by 220
“As a data-guided simulation using known gene sets [21,25], we introduce an algorithm for using real data in the Results section.”
Gene set analysis of genome-wide association studies: Methodological issues and perspectives
Wang, L., Jia, P., Wolfinger, R. D., Chen, X., & Zhao, Z. (2011) · Genomics · doi:10.1016/j.ygeno.2011.04.006 · Cited by 207
A comparison of gene set analysis methods in terms of sensitivity, prioritization and specificity
Tarca, A. L., Bhatti, G., & Romero, R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0079217 · Cited by 203
Mondul, A. M., Moore, S. C., Weinstein, S. J., Karoly, E. D., Sampson, J. N., & Albanes, D. (2015) · International Journal of Cancer · doi:10.1002/ijc.29576 · Cited by 179
ChIP-Enrich: gene set enrichment testing for ChIP-seq data
Welch, R. P., Lee, C., Imbriano, P. M., Patil, S., Weymouth, T. E., Smith, R. A., Scott, L. J., & Sartor, M. A. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku463 · Cited by 151
Zhang, Z., Schlamp, F., Huang, L., Clark, H., & Brayboy, L. (2020) · Reproduction · doi:10.1530/rep-19-0330 · Cited by 150
“Gene Set analysis was done using the GSA R-package (Efron & Tibshirani 2007).”
MethodMulti‐omic rejuvenation of naturally aged tissues by a single cycle of transient reprogramming
Chondronasiou, D., Gill, D., Mosteiro, L., Urdinguio, R. G., Berenguer‐Llergo, A., Aguilera, M., Durand, S., Aprahamian, F., Nirmalathasan, N., Abad, M., et al. (2022) · Aging Cell · doi:10.1111/acel.13578 · Cited by 133
Pearson, B. L., Simon, J. M., McCoy, E. S., Salazar, G., Fragola, G., & Zylka, M. J. (2016) · Nature Communications · doi:10.1038/ncomms11173 · Cited by 127
Choi, W., Miyamura, Y., Wolber, R., Smuda, C., Reinhold, W., Liu, H., Kolbe, L., & Hearing, V. J. (2010) · Journal of Investigative Dermatology · doi:10.1038/jid.2010.5 · Cited by 117
Pesticides applied to crops and amyotrophic lateral sclerosis risk in the U.S
Andrew, A., Zhou, J., Gui, J., Harrison, A., Shi, X., Li, M., Guetti, B., Nathan, R., Tischbein, M., Pioro, E. P., et al. (2021) · NeuroToxicology · doi:10.1016/j.neuro.2021.09.004 · Cited by 54
Guidelines for using sigQC for systematic evaluation of gene signatures
Dhawan, A., Barberis, A., Cheng, W.-C., Domingo, E., West, C., Maughan, T., Scott, J. G., Harris, A. L., & Buffa, F. M. (2019) · Nature Protocols · doi:10.1038/s41596-019-0136-8 · Cited by 31
Backes, C., Rühle, F., Stoll, M., Haas, J., Frese, K., Franke, A., Lieb, W., Wichmann, H.-E., Weis, T., Kloos, W., et al. (2014) · BMC Genomics · doi:10.1186/1471-2164-15-622 · Cited by 26
“A similar claim has already been made by Efron and Thibsirani [32] who address the problem of identifying differentially expressed groups of genes from a microarray experiment based on the gene set enrichment analysis (GSEA).”
BackgroundCardiovascular Risk and Resilience Among Black Adults: Rationale and Design of the MECA Study
Islam, S. J., Kim, J. H., Topel, M., Liu, C., Ko, Y., Mujahid, M. S., Sims, M., Mubasher, M., Ejaz, K., Morgan‐Billingslea, J., et al. (2020) · Journal of the American Heart Association · doi:10.1161/jaha.119.015247 · Cited by 23
PSEA: Phenotype Set Enrichment Analysis—A New Method for Analysis of Multiple Phenotypes
Ried, J. S., Döring, A., Oexle, K., Meisinger, C., Winkelmann, J., Klopp, N., Meitinger, T., Peters, A., Suhre, K., Wichmann, H. ‐., et al. (2012) · Genetic Epidemiology · doi:10.1002/gepi.21617 · Cited by 21
“This is a modification for positive test statistics of the widely used maxmean statistic [Efron and Tibshirani, 2007], which was presented as sumstat score for gene set enrichment [Tintle et al.”
MethodInfluentialNavigating Transcriptomic Connectivity Mapping Workflows to Link Chemicals with Bioactivities
Shah, I., Bundy, J., Chambers, B., Everett, L. J., Haggard, D., Harrill, J., Judson, R. S., Nyffeler, J., & Patlewicz, G. (2022) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.2c00245 · Cited by 19
Metadata
- Authors
- Efron, B., & Tibshirani, R.
- Venue
- The Annals of Applied Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2007-06-01
- Volume
- 1
- Issue
- 1
- Pages
- 107-129
- ISSN
- 1932-6157 (ISSN-L)
- OpenAlex
- W2020859582
- Cited by
- 798
- References
- 20
- FWCI
- 20.11
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.