CoChat

Features that define the best ChIP-seq peak calling algorithms

Open access
Thomas, Reuben, Thomas, Sean, Holloway, Alisha K., Pollard, Katherine S. · 2016 · Briefings in Bioinformatics

Published version of a preprint

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Abstract

Chromatin immunoprecipitation followed by sequencing (ChIP-seq) is an important tool for studying gene regulatory proteins, such as transcription factors and histones. Peak calling is one of the first steps in the analysis of these data. Peak calling consists of two sub-problems: identifying candidate peaks and testing candidate peaks for statistical significance. We surveyed 30 methods and identified 12 features of the two sub-problems that distinguish methods from each other. We picked six methods GEM, MACS2, MUSIC, BCP, Threshold-based method (TM) and ZINBA] that span this feature space and used a combination of 300 simulated ChIP-seq data sets, 3 real data sets and mathematical analyses to identify features of methods that allow some to perform better than the others. We prove that methods that explicitly combine the signals from ChIP and input samples are less powerful than methods that do not. Methods that use windows of different sizes are more powerful than the ones that do not. For statistical testing of candidate peaks, methods that use a Poisson test to rank their candidate peaks are more powerful than those that use a Binomial test. BCP and MACS2 have the best operating characteristics on simulated transcription factor binding data. GEM has the highest fraction of the top 500 peaks containing the binding motif of the immunoprecipitated factor, with 50% of its peaks within 10 base pairs of a motif. BCP and MUSIC perform best on histone data. These findings provide guidance and rationale for selecting the best peak caller for a given application.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bib/bbw035

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Quantile–quantile plots comparing the distributions of reads in the input and ChIP sample in one of the simulated data sets with those in a real Tbx5 ChIP sample from mouse cardiomyocytes. The dotted lines represent the linear fits to the data excluding the one extreme point in both (A) and (B). The pearson correlation of the scatter of points modeled by the dotted lines is 0.99 in (A) and 0.97 in (B).
Figure 1. Quantile–quantile plots comparing the distributions of reads in the input and ChIP sample in one of the simulated data sets with those in a real Tbx5 ChIP sample from mouse cardiomyocytes. The dotted lines represent the linear fits to the data excluding the one extreme point in both (A) and (B). The pearson correlation of the scatter of points modeled by the dotted lines is 0.99 in (A) and 0.97 in (B).CC BY-NC · PubMed Central
Figure 2. Sensitivity (a), Precision (b) and F-score (c) as a function of the log10 of the number of called peaks for the six peak calling methods on 100 simulated transcription factor ChIP-seq data sets under the medium noise setting. For each method, the means and 95% confidence intervals (dark gray regions around the mean profiles) of the means of a metric are estimated using Generalized Additive Models [51] of variation across the 100 simulated data sets, as a function of a smooth function of log10 of the number of called peaks. Note the overlapping performances of GEM and MUSIC and BCP and MACS2 in (a), (b) and (c) with log10 number of peaks, respectively, between 1.2 and 1.6 and 1.6 and 2.0.
Figure 2. Sensitivity (a), Precision (b) and F-score (c) as a function of the log10 of the number of called peaks for the six peak calling methods on 100 simulated transcription factor ChIP-seq data sets under the medium noise setting. For each method, the means and 95% confidence intervals (dark gray regions around the mean profiles) of the means of a metric are estimated using Generalized Additive Models [51] of variation across the 100 simulated data sets, as a function of a smooth function of log10 of the number of called peaks. Note the overlapping performances of GEM and MUSIC and BCP and MACS2 in (a), (b) and (c) with log10 number of peaks, respectively, between 1.2 and 1.6 and 1.6 and 2.0.CC BY-NC · PubMed Central
Figure 3. Median distance-binding (a) and Median distance-peak (b) as a function of the log10 of the number of called peaks for the six peak calling methods on 100 simulated data sets under the medium noise setting. The means and 95% confidence intervals are estimated in a similar manner as is done for Figure 2. Note the overlapping performances of MACS2 and MUSIC in (a), GEM, MUSIC and TM in (b) with log10 number of peaks, respectively, between 1.6 and 2.0 and 1.2 and 2.0.
Figure 3. Median distance-binding (a) and Median distance-peak (b) as a function of the log10 of the number of called peaks for the six peak calling methods on 100 simulated data sets under the medium noise setting. The means and 95% confidence intervals are estimated in a similar manner as is done for Figure 2. Note the overlapping performances of MACS2 and MUSIC in (a), GEM, MUSIC and TM in (b) with log10 number of peaks, respectively, between 1.6 and 2.0 and 1.2 and 2.0.CC BY-NC · PubMed Central
Figure 4. Fraction of top n peaks within 100 bp of a Tbx5 motif for the six methods is given. Results are based on Tbx5 Motif 1.
Figure 4. Fraction of top n peaks within 100 bp of a Tbx5 motif for the six methods is given. Results are based on Tbx5 Motif 1.CC BY-NC · PubMed Central
Figure 5. Empirical distribution of the shortest distance to the Tbx5 motif of the significant peaks called by the six methods. Results based on Tbx5 Motif 1. Note the overlapping performances of BCP, MACS2 and TM over the entire range and of GEM, BCP, MACS2 and TM with distance between 100 and 10 000 bp.
Figure 5. Empirical distribution of the shortest distance to the Tbx5 motif of the significant peaks called by the six methods. Results based on Tbx5 Motif 1. Note the overlapping performances of BCP, MACS2 and TM over the entire range and of GEM, BCP, MACS2 and TM with distance between 100 and 10 000 bp.CC BY-NC · PubMed Central
Figure 6. Sensitivity (A), precision (B) and F-score (C) of the overlap of the called significant peak regions with active gene bodies for H3K36me3 data. The threshold for defining active genes was varied from 0 to 2 RPKM. Note the overlapping performances of MACS2 and ZINBA in (A), MACS2 and TM in (B) and BCP and ZINBA in (B).
Figure 6. Sensitivity (A), precision (B) and F-score (C) of the overlap of the called significant peak regions with active gene bodies for H3K36me3 data. The threshold for defining active genes was varied from 0 to 2 RPKM. Note the overlapping performances of MACS2 and ZINBA in (A), MACS2 and TM in (B) and BCP and ZINBA in (B).CC BY-NC · PubMed Central
Figure 7. Correct peak fraction (fraction of top called 1000n peaks that overlap with the promoters of active genes, genes with expression > 0.5 RPKM) as a function of recovered promoter fraction (fraction of promoters of active genes that overlap with the top called 1000n peaks) for H3K4me3 data is given. The peaks for each method were ranked by their assigned significance.
Figure 7. Correct peak fraction (fraction of top called 1000n peaks that overlap with the promoters of active genes, genes with expression > 0.5 RPKM) as a function of recovered promoter fraction (fraction of promoters of active genes that overlap with the top called 1000n peaks) for H3K4me3 data is given. The peaks for each method were ranked by their assigned significance.CC BY-NC · PubMed Central
Figure 8. Type I error rate and statistical power comparison between Poisson and Binomial tests is given. f is a parameter that controls the increase in the proportion of DNA from a given region in the input relative to the ChIP sample for the Type I error evaluations, (A), (B) and (C), and increase in this proportion for the ChIP relative to the input sample for the power evaluations ((D) and (E)). The dashed lines in (A), (B) and (C) are the y = x lines.
Figure 8. Type I error rate and statistical power comparison between Poisson and Binomial tests is given. f is a parameter that controls the increase in the proportion of DNA from a given region in the input relative to the ChIP sample for the Type I error evaluations, (A), (B) and (C), and increase in this proportion for the ChIP relative to the input sample for the power evaluations ((D) and (E)). The dashed lines in (A), (B) and (C) are the y = x lines.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 50 references

Research works printed but not linked: 2

  1. Bhattachayya A. On a measure of divergence between two statistical population defined by their population distributions. Bull Cal Math Soc 1943; 35: 99– 109.
  2. Durrett R. Probability Theories and Examples. Duxbury Press, 1996.

Cited by

See all 112

Metadata

Authors
Thomas, R., Thomas, S., Holloway, A. K., & Pollard, K. S.
Affiliations
Gladstone Institutes (US)
Corresponding author
Thomas, Reuben
Venue
Briefings in Bioinformatics
Published
2016-05-11
Volume
18
Issue
3
Pages
bbw035
ISSN
  • 1467-5463 (ISSN-L)
  • 1477-4054
OpenAlex
W2385696187
Cited by
112
References
49
Citations by year
2026: 2 · 2025: 8 · 2024: 5 · 2023: 10 · 2022: 13 · 2021: 20 · 2020: 15 · 2019: 15 · 2018: 12 · 2017: 10 · 2016: 2
FWCI
6.72
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Briefings in Bioinformatics (publishedVersion, cc-by-nc)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • Gladstone Institutes
  • National Heart, Lung, and Blood Institute (HL098179, UM1HL098179, P01HL089707, HL089707, U01HL098179)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
ChIP-seq, peak calling, statistical significance testing, binomial test, transcription factor binding, histone modifications, MACS2, gems, BCP
MeSH terms
Algorithms, Binding Sites, Histones, Transcription Factors, Sequence Analysis, DNA, Oligonucleotide Array Sequence Analysis, Chromatin Immunoprecipitation, High-Throughput Nucleotide Sequencing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.