The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Measuring reproducibility of high-throughput experiments
Open accessAbstract
Reproducibility is essential to reliable scientific discovery in high-throughput experiments. In this work we propose a unified approach to measure the reproducibility of findings identified from replicate experiments and identify putative discoveries using reproducibility. Unlike the usual scalar measures of reproducibility, our approach creates a curve, which quantitatively assesses when the findings are no longer consistent across replicates. Our curve is fitted by a copula mixture model, from which we derive a quantitative reproducibility score, which we call the “irreproducible discovery rate” (IDR) analogous to the FDR. This score can be computed at each set of paired replicate ranks and permits the principled setting of thresholds both for assessing reproducibility and combining replicates. Since our approach permits an arbitrary scale for each replicate, it provides useful descriptive measures in a wide variety of situations to be explored. We study the performance of the algorithm using simulations and give a heuristic analysis of its theoretical properties. We demonstrate the effectiveness of our method in a ChIP-seq experiment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Mapping and quantifying mammalian transcriptomes by RNA-Seq
Mortazavi, A., Williams, B. A., McCue, K., Schaeffer, L., & Wold, B. (2008) · Nature Methods · doi:10.1038/nmeth.1226 · Cited by 13,696
ChIP–seq: advantages and challenges of a maturing technology
Park, P. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2641 · Cited by 2,217
MAQC Consortium, Shi, L., Shi, L., Reid, L. H., Jones, W. D., Shippy, R., Warrington, J. A., Baker, S. C., Collins, P. J., de Longueville, F., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1239 · Cited by 2,126
Self-citation
Cited by
An integrated encyclopedia of DNA elements in the human genome
The ENCODE Project Consortium (2012) · Nature · doi:10.1038/nature11247 · Cited by 18,072
Landscape of transcription in human cells
Djebali, S., Davis, C. A., Merkel, A., Dobin, A., Lassmann, T., Mortazavi, A., Tanzer, A., Lagarde, J., Lin, W., Schlesinger, F., et al. (2012) · Nature · doi:10.1038/nature11233 · Cited by 5,187
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530
ChIP-seq guidelines and practices of the ENCODE and modENCODE consortia
Landt, S. G., Marinov, G. K., Kundaje, A., Kheradpour, P., Pauli, F., Batzoglou, S., Bernstein, B. E., Bickel, P., Brown, J. B., Cayting, P., et al. (2012) · Genome Research · doi:10.1101/gr.136184.111 · Cited by 2,033
Self-citationRobust transcriptome-wide discovery of RNA-binding protein binding sites with enhanced CLIP (eCLIP)
Van Nostrand, E. L., Pratt, G. A., Shishkin, A. A., Gelboin-Burkhart, C., Fang, M. Y., Sundararaman, B., Blue, S. M., Nguyen, T. B., Surka, C., Elkins, K., et al. (2016) · Nature Methods · doi:10.1038/nmeth.3810 · Cited by 1,397
A large-scale binding and functional map of human RNA-binding proteins
Van Nostrand, E. L., Freese, P., Pratt, G. A., Wang, X., Wei, X., Xiao, R., Blue, S. M., Chen, J.-Y., Cody, N. A. L., Dominguez, D., et al. (2020) · Nature · doi:10.1038/s41586-020-2077-3 · Cited by 1,317
Genome-wide association between YAP/TAZ/TEAD and AP-1 at enhancers drives oncogenic growth
Zanconato, F., Forcato, M., Battilana, G., Azzolin, L., Quaranta, E., Bodega, B., Rosato, A., Bicciato, S., Cordenonsi, M., & Piccolo, S. (2015) · Nature Cell Biology · doi:10.1038/ncb3216 · Cited by 1,219
Palantir characterizes cell fate continuities in human hematopoiesis
Setty, M., Kiseliovas, V., Levine, J., Gayoso, A., Mazutis, L., & Pe’er, D. (2018) · doi:10.1101/385328 · Cited by 836
Characterization of cell fate probabilities in single-cell data with Palantir
Setty, M., Kiseliovas, V., Levine, J., Gayoso, A., Mazutis, L., & Pe’er, D. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0068-4 · Cited by 833
Aerobic glycolysis promotes T helper 1 cell differentiation through an epigenetic mechanism
Peng, M., Yin, N., Chhangawala, S., Xu, K., Leslie, C. S., & Li, M. O. (2016) · Science · doi:10.1126/science.aaf6284 · Cited by 787
Brain cell type–specific enhancer–promoter interactome maps and disease - risk association
Nott, A., Holtman, I. R., Coufal, N. G., Schlachetzki, J. C. M., Yu, M., Hu, R., Han, C. Z., Pena, M., Xiao, J., Wu, Y., et al. (2019) · Science · doi:10.1126/science.aay0793 · Cited by 699
Widespread plasticity in CTCF occupancy linked to DNA methylation
Wang, H., Maurano, M. T., Qu, H., Varley, K. E., Gertz, J., Pauli, F., Lee, K., Canfield, T., Weaver, M., Sandstrom, R., et al. (2012) · Genome Research · doi:10.1101/gr.136101.111 · Cited by 588
“The set of peaks reproducible in both replicates were identified based on an irreproducible discovery rate (IDR) of 0.25% (Li et al. 2011).”
InfluentialSystematic mapping of functional enhancer–promoter connections with CRISPR interference
Fulco, C. P., Munschauer, M., Anyoha, R., Munson, G., Grossman, S. R., Perez, E. M., Kane, M., Cleary, B., Lander, E. S., & Engreitz, J. M. (2016) · Science · doi:10.1126/science.aag2445 · Cited by 643
Base-resolution models of transcription-factor binding reveal soft motif syntax
Avsec, Ž., Weilert, M., Shrikumar, A., Krueger, S., Alexandari, A., Dalal, K., Fropf, R., McAnany, C., Gagneur, J., Kundaje, A., et al. (2021) · Nature Genetics · doi:10.1038/s41588-021-00782-6 · Cited by 610
A new view of transcriptome complexity and regulation through the lens of local splicing variations
Vaquero-Garcia, J., Barrera, A., Gazzara, M. R., González-Vallinas, J., Lahens, N. F., Hogenesch, J. B., Lynch, K. W., & Barash, Y. (2016) · eLife · doi:10.7554/elife.11752 · Cited by 488
Seidman, J. S., Troutman, T. D., Sakai, M., Gola, A., Spann, N. J., Bennett, H., Bruni, C. M., Ouyang, Z., Li, R. Z., Sun, X., et al. (2020) · Immunity · doi:10.1016/j.immuni.2020.04.001 · Cited by 443
Enhancer-promoter interactions are encoded by complex genomic signatures on looping chromatin
Whalen, S., Truty, R. M., & Pollard, K. S. (2016) · Nature Genetics · doi:10.1038/ng.3539 · Cited by 456
Sakai, M., Troutman, T. D., Seidman, J. S., Ouyang, Z., Spann, N. J., Abe, Y., Ego, K. M., Bruni, C. M., Deng, Z., Schlachetzki, J. C. M., et al. (2019) · Immunity · doi:10.1016/j.immuni.2019.09.002 · Cited by 418
The SWI/SNF chromatin remodelling complex is required for maintenance of lineage specific enhancers
Alver, B. H., Kim, K. H., Lu, P., Wang, X., Manchester, H. E., Wang, W., Haswell, J. R., Park, P. J., & Roberts, C. W. M. (2017) · Nature Communications · doi:10.1038/ncomms14648 · Cited by 427
CHiCAGO: robust detection of DNA looping interactions in Capture Hi-C data
Cairns, J., Freire-Pritchett, P., Wingett, S. W., Várnai, C., Dimond, A., Plagnol, V., Zerbino, D., Schoenfelder, S., Javierre, B.-M., Osborne, C., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-0992-2 · Cited by 417
Single-cell mapping of the thymic stroma identifies IL-25-producing tuft epithelial cells
Bornstein, C., Nevo, S., Giladi, A., Kadouri, N., Pouzolles, M., Gerbe, F., David, E., Machado, A., Chuprin, A., Tóth, B., et al. (2018) · Nature · doi:10.1038/s41586-018-0346-1 · Cited by 317
Iterative random forests to discover predictive and stable high-order interactions
Basu, S., Kumbier, K., Brown, J. B., & Yu, B. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1711236115 · Cited by 317
Self-citationRobertson, C. C., Inshaw, J. R. J., Onengut-Gumuscu, S., Chen, W.-M., Santa Cruz, D. F., Yang, H., Cutler, A. J., Crouch, D. J. M., Farber, E., Bridges, S. L., et al. (2021) · Nature Genetics · doi:10.1038/s41588-021-00880-5 · Cited by 302
Role of DNA Methylation in Modulating Transcription Factor Occupancy
Maurano, M. T., Wang, H., John, S., Shafer, A., Canfield, T., Lee, K., & Stamatoyannopoulos, J. A. (2015) · Cell Reports · doi:10.1016/j.celrep.2015.07.024 · Cited by 281
Heterogeneity in old fibroblasts is linked to variability in reprogramming and wound healing
Mahmoudi, S., Mancini, E., Xu, L., Moore, A., Jahanbani, F., Hebestreit, K., Srinivasan, R., Li, X., Devarajan, K., Prélot, L., et al. (2019) · Nature · doi:10.1038/s41586-019-1658-5 · Cited by 285
Hotspot identifies informative gene modules across modalities of single-cell genomics
DeTomaso, D., & Yosef, N. (2021) · Cell Systems · doi:10.1016/j.cels.2021.04.005 · Cited by 267
Functional Genetic Variants Revealed by Massively Parallel Precise Genome Editing
Sharon, E., Chen, S.-A. A., Khosla, N. M., Smith, J. D., Pritchard, J. K., & Fraser, H. B. (2018) · Cell · doi:10.1016/j.cell.2018.08.057 · Cited by 225
Lineage-Determining Transcription Factor TCF-1 Initiates the Epigenetic Identity of T Cells
Johnson, J. L., Georgakilas, G., Petrovic, J., Kurachi, M., Cai, S., Harly, C., Pear, W. S., Bhandoola, A., Wherry, E. J., & Vahedi, G. (2018) · Immunity · doi:10.1016/j.immuni.2018.01.012 · Cited by 226
Rewiring of human neurodevelopmental gene regulatory programs by human accelerated regions
Girskis, K. M., Stergachis, A. B., DeGennaro, E. M., Doan, R. N., Qian, X., Johnson, M. B., Wang, P. P., Sejourne, G. M., Nagy, M. A., Pollina, E. A., et al. (2021) · Neuron · doi:10.1016/j.neuron.2021.08.005 · Cited by 193
Link, V. M., Duttke, S. H., Chun, H. B., Holtman, I. R., Westin, E., Hoeksema, M. A., Abe, Y., Skola, D., Romanoski, C. E., Tao, J., et al. (2018) · Cell · doi:10.1016/j.cell.2018.04.018 · Cited by 182
Sequence determinants of human gene regulatory elements
Sahu, B., Hartonen, T., Pihlajamaa, P., Wei, B., Dave, K., Zhu, F., Kaasinen, E., Lidschreiber, K., Lidschreiber, M., Daub, C. O., et al. (2022) · Nature Genetics · doi:10.1038/s41588-021-01009-4 · Cited by 198
Spatially resolved epigenomic profiling of single cells in complex tissues
Lu, T., Ang, C. E., & Zhuang, X. (2022) · Cell · doi:10.1016/j.cell.2022.09.035 · Cited by 197
Girdhar, K., Hoffman, G. E., Jiang, Y., Brown, L., Kundakovic, M., Hauberg, M. E., Francoeur, N. J., Wang, Y.-C., Shah, H., Kavanagh, D. H., et al. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0187-0 · Cited by 139
Bonhoure, N., Bounova, G., Bernasconi, D., Praz, V., Lammers, F., Canella, D., Willis, I. M., Herr, W., Hernandez, N., Delorenzi, M., et al. (2014) · Genome Research · doi:10.1101/gr.168260.113 · Cited by 155
“Scaling to the total amount of sequence tags that can be aligned onto the genome is usually the first step (for examples, see Li et al. 2011; Landt et al. 2012; Le Martelot et al. 2012).”
BackgroundDeletions of chromosomal regulatory boundaries are associated with congenital disease
Ibn-Salem, J., Köhler, S., Love, M. I., Chung, H.-R., Huang, N., Hurles, M. E., Haendel, M., Washington, N. L., Smedley, D., Mungall, C. J., et al. (2014) · Genome Biology · doi:10.1186/s13059-014-0423-1 · Cited by 143
Cx3cr1- deficient microglia exhibit a premature aging transcriptome
Gyoneva, S., Hosur, R., Gosselin, D., Zhang, B., Ouyang, Z., Cotleur, A. C., Peterson, M., Allaire, N., Challa, R., Cullen, P., et al. (2019) · Life Science Alliance · doi:10.26508/lsa.201900453 · Cited by 103
“Moreover, Tnf , a classical in fl ammatory gene, had higher expression in Cx3cr1 - KO microglia by RNA-seq and a trend for differential Rbp2 binding by ChIP-seq (irreproducible discovery rate [IDR] = 0.0553; Fig 2D).”
InfluentialBaranasic, D., Hörtenhuber, M., Balwierz, P. J., Zehnder, T., Mukarram, A. K., Nepal, C., Várnai, C., Hadzhiev, Y., Jimenez-Gonzalez, A., Li, N., et al. (2022) · Nature Genetics · doi:10.1038/s41588-022-01089-w · Cited by 101
Engineering a haematopoietic stem cell niche by revitalizing mesenchymal stromal cells
Nakahara, F., Borger, D. K., Wei, Q., Pinho, S., Maryanovich, M., Zahalka, A. H., Suzuki, M., Cruz, C. D., Wang, Z., Xu, C., et al. (2019) · Nature Cell Biology · doi:10.1038/s41556-019-0308-3 · Cited by 99
Prediction of enhancer-promoter interactions via natural language processing
Zeng, W., Wu, M., & Jiang, R. (2018) · BMC Genomics · doi:10.1186/s12864-018-4459-6 · Cited by 96
“Using GENCODE [29] version 19 annotations and RNA-seq data from ENCODE, promoters were reserved if actively transcribed (mean FPKM > 0.3 [30] with irreproducible discovery rate 0.1 [31]) in each cell line.”
MethodInfluentialGenetic Regulatory Mechanisms of Smooth Muscle Cells Map to Coronary Artery Disease Risk Loci
Liu, B., Pjanic, M., Wang, T., Nguyen, T., Gloudemans, M., Rao, A., Castano, V. G., Nurnberg, S., Rader, D. J., Elwyn, S., et al. (2018) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2018.08.001 · Cited by 94
Pelikan, R. C., Kelly, J. A., Fu, Y., Lareau, C. A., Tessneer, K. L., Wiley, G. B., Wiley, M. M., Glenn, S. B., Harley, J. B., Guthridge, J. M., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-05328-9 · Cited by 94
SALL1 enforces microglia-specific DNA binding and function of SMADs to establish microglia identity
Fixsen, B. R., Han, C. Z., Zhou, Y., Spann, N. J., Saisan, P., Shen, Z., Balak, C., Sakai, M., Cobo, I., Holtman, I. R., et al. (2023) · Nature Immunology · doi:10.1038/s41590-023-01528-8 · Cited by 87
ZBTB33 binds unmethylated regions of the genome associated with actively expressed genes
Blattler, A., Yao, L., Wang, Y., Ye, Z., Jin, V. X., & Farnham, P. J. (2013) · Epigenetics & Chromatin · doi:10.1186/1756-8935-6-13 · Cited by 82
Miller, A., Ralser, M., Kloet, S. L., Loos, R., Nishinakamura, R., Bertone, P., Vermeulen, M., & Hendrich, B. (2016) · Development · doi:10.1242/dev.139113 · Cited by 78
“Bioinformatic analyses Sall4-FLAG, Mbd3 and Chd4 ChIP-seq data were analysed using the irreproducible discovery rate (IDR) method, which assesses replicate agreement and therefore only calls peaks that are strong in all replicates (Landt et al., 2012; Li et al., 2011).”
MethodReproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
Beyond antioxidant genes in the ancient Nrf2 regulatory network
Lacher, S. E., Lee, J. S., Wang, X., Campbell, M. R., Bell, D. A., & Slattery, M. (2015) · Free Radical Biology and Medicine · doi:10.1016/j.freeradbiomed.2015.06.044 · Cited by 77
TBR1 regulates autism risk genes in the developing neocortex
Notwell, J. H., Heavner, W. E., Darbandi, S. F., Katzman, S., McKenna, W. L., Ortiz-Londono, C. F., Tastad, D., Eckler, M. J., Rubenstein, J. L. R., McConnell, S. K., et al. (2016) · Genome Research · doi:10.1101/gr.203612.115 · Cited by 74
Pro-neuronal activity of Myod1 due to promiscuous binding to neuronal genes
Lee, Q. Y., Mall, M., Chanda, S., Zhou, B., Sharma, K. S., Schaukowitch, K., Adrian-Segarra, J. M., Grieder, S. D., Kareta, M. S., Wapinski, O. L., et al. (2020) · Nature Cell Biology · doi:10.1038/s41556-020-0490-3 · Cited by 50
Wang, X., Campbell, M. R., Lacher, S. E., Cho, H.-Y., Wan, M., Crowl, C. L., Chorley, B. N., Bond, G. L., Kleeberger, S. R., Slattery, M., et al. (2016) · Cell Reports · doi:10.1016/j.celrep.2016.03.068 · Cited by 49
Scalable and deep profiling of mRNA targets for individual microRNAs with chimeric eCLIP
Manakov, S. A., Shishkin, A. A., Yee, B. A., Shen, K. A., Cox, D. C., Park, S. S., Foster, H. M., Chapman, K. B., Yeo, G. W., & Van Nostrand, E. L. (2022) · doi:10.1101/2022.02.13.480296 · Cited by 43
Dubois-Chevalier, J., Dubois, V., Dehondt, H., Mazrooei, P., Mazuy, C., Sérandour, A. A., Gheeraert, C., Guillaume, P., Baugé, E., Derudas, B., et al. (2017) · Genome Research · doi:10.1101/gr.217075.116 · Cited by 29
Haensel, D., Daniel, B., Gaddam, S., Pan, C., Fabo, T., Bjelajac, J., Jussila, A. R., Gonzalez, F., Li, N. Y., Chen, Y., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-37993-w · Cited by 29
A phage nucleus-associated RNA-binding protein is required for jumbo phage infection
Enustun, E., Armbruster, E. G., Lee, J., Zhang, S., Yee, B. A., Malukhina, K., Gu, Y., Deep, A., Naritomi, J. T., Liang, Q., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae216 · Cited by 27
Hiatt, S. M., Trajkova, S., Sebastiano, M. R., Partridge, E. C., Abidi, F. E., Anderson, A., Ansar, M., Antonarakis, S. E., Azadi, A., Bachmann-Gagescu, R., et al. (2023) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2022.12.007 · Cited by 21
Helios represses megakaryocyte priming in hematopoietic stem and progenitor cells
Cova, G., Taroni, C., Deau, M.-C., Cai, Q., Mittelheisser, V., Philipps, M., Jung, M., Cerciat, M., Le Gras, S., Thibault-Carpentier, C., et al. (2021) · Journal of Experimental Medicine · doi:10.1084/jem.20202317 · Cited by 17
Li, H., & Guan, Y. (2019) · doi:10.1101/856823 · Cited by 3
PRIME-BSPre: a genome-wide protein-RNA binding sites prediction method based on templates
Wei, X., Duan, Y., Li, D., Liu, X., Xie, J., & Liu, S. (2026) · BMC Genomics · doi:10.1186/s12864-026-12657-3
Metadata
- Authors
- Li, Q., Brown, J. B., Huang, H., & Bickel, P. J.
- Venue
- The Annals of Applied Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2011-09-01
- Volume
- 5
- Issue
- 3
- Pages
- 1752-1779
- ISSN
- 1932-6157 (ISSN-L)
- OpenAlex
- W2081942683
- Cited by
- 1,009
- References
- 5
- FWCI
- 8.82
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
