Genomic diversity affects the accuracy of bacterial single-nucleotide polymorphism–calling pipelines
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Background: Accurately identifying single-nucleotide polymorphisms (SNPs) from bacterial sequencing data is an essential requirement for using genomics to track transmission and predict important phenotypes such as antimicrobial resistance. However, most previous performance evaluations of SNP calling have been restricted to eukaryotic (human) data. Additionally, bacterial SNP calling requires choosing an appropriate reference genome to align reads to, which, together with the bioinformatic pipeline, affects the accuracy and completeness of a set of SNP calls obtained. This study evaluates the performance of 209 SNP-calling pipelines using a combination of simulated data from 254 strains of 10 clinically common bacteria and real data from environmentally sourced and genomically diverse isolates within the genera Citrobacter, Enterobacter, Escherichia, and Klebsiella. Results: We evaluated the performance of 209 SNP-calling pipelines, aligning reads to genomes of the same or a divergent strain. Irrespective of pipeline, a principal determinant of reliable SNP calling was reference genome selection. Across multiple taxa, there was a strong inverse relationship between pipeline sensitivity and precision, and the Mash distance (a proxy for average nucleotide divergence) between reads and reference genome. The effect was especially pronounced for diverse, recombinogenic bacteria such as Escherichia coli but less dominant for clonal species such as Mycobacterium tuberculosis. Conclusions: The accuracy of SNP calling for a given species is compromised by increasing intra-species diversity. When reads were aligned to the same genome from which they were sequenced, among the highest-performing pipelines was Novoalign/GATK. By contrast, when reads were aligned to particularly divergent genomes, the highest-performing pipelines often used the aligners NextGenMap or SMALT, and/or the variant callers LoFreq, mpileup, or Strelka.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Minimap2: Pairwise alignment for nucleotide sequences
Li, H. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty191 · Cited by 14,807
O'Leary, N. A., Wright, M. W., Brister, J. R., Ciufo, S., Haddad, D., McVeigh, R., Rajput, B., Robbertse, B., Smith-White, B., Ako-Adjei, D., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1189 · Cited by 6,357
High throughput ANI analysis of 90K prokaryotic genomes reveals clear species boundaries
Jain, C., Rodriguez-R, L. M., Phillippy, A. M., Konstantinidis, K. T., & Aluru, S. (2018) · Nature Communications · doi:10.1038/s41467-018-07641-9 · Cited by 5,748
Roary: rapid large-scale prokaryote pan genome analysis
Page, A. J., Cummins, C. A., Hunt, M., Wong, V. K., Reuter, S., Holden, M. T. G., Fookes, M., Falush, D., Keane, J. A., & Parkhill, J. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv421 · Cited by 5,736
Kraken: ultrafast metagenomic sequence classification using exact alignments
Wood, D. E., & Salzberg, S. L. (2014) · Genome Biology · doi:10.1186/gb-2014-15-3-r46 · Cited by 4,521 · cited 1×
“0.7 (Kraken, RRID:SCR 005484) [125], respectively.”
InfluentialHaplotype-based variant detection from short-read sequencing
Garrison, E., & Marth, G. (2012) · arXiv · doi:10.48550/arxiv.1207.3907 · Cited by 3,700
Mash: fast genome and metagenome distance estimation using MinHash
Ondov, B. D., Treangen, T. J., Melsted, P., Mallonee, A. B., Bergman, N. H., Koren, S., & Phillippy, A. M. (2016) · Genome biology · doi:10.1186/s13059-016-0997-x · Cited by 3,373
MUMmer4: A fast and versatile genome alignment system
Marçais, G., Delcher, A. L., Phillippy, A. M., Coston, R., Salzberg, S. L., & Zimin, A. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005944 · Cited by 2,484
A universal SNP and small-indel variant caller using deep neural networks
Poplin, R., Chang, P.-C., Alexander, D., Schwartz, S., Colthurst, T., Ku, A., Newburger, D., Dijamco, J., Nguyen, N., Afshar, P. T., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4235 · Cited by 1,840 · cited 2×
“…Stampy [38] [both with and without pre-alignment with BWA-aln], and Yara [39]) used in conjunction with 14 variant callers (16GT [40], DeepVariant [41], Freebayes [42], GATK HaplotypeCaller [24, 25], LoFreq [43], mpileup [20], Octopus [44], Pilon [45], Platypus [46], SolSNP [47], SNVer [48],…”
Genome structural variation discovery and genotyping
Alkan, C., Coe, B. P., & Eichler, E. E. (2011) · Nature Reviews Genetics · doi:10.1038/nrg2958 · Cited by 1,630 · cited 1×
“For example, up to 5% of SNPs are uniquely called by 1 of 5 different pipelines [9] with even lower agreement on structural variants [10].”
Centrifuge: rapid and sensitive classification of metagenomic sequences
Kim, D., Song, L., Breitwieser, F. P., & Salzberg, S. L. (2016) · Genome Research · doi:10.1101/gr.210641.116 · Cited by 1,438 · cited 1×
“If assuming all input reads are from a single, unknown, origin, then in principle a reference genome could be predicted using a metagenomic classifier such as Centrifuge [86], CLARK [87], Kaiju [88], or Kraken [89].”
Stampy: A statistical algorithm for sensitive and fast mapping of Illumina sequence reads
Lunter, G., & Goodson, M. (2011) · Genome Research · doi:10.1101/gr.111120.110 · Cited by 1,016
A survey of sequence alignment algorithms for next-generation sequencing
Li, H., & Homer, N. (2010) · Briefings in Bioinformatics · doi:10.1093/bib/bbq015 · Cited by 875
Database Resources of the National Center for Biotechnology Information
Coordinators, N. R. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1071 · Cited by 817
Extensive sequencing of seven human genomes to characterize benchmark reference materials
Zook, J. M., Catoe, D., McDaniel, J., Vang, L., Spies, N., Sidow, A., Weng, Z., Liu, Y., Mason, C. E., Alexander, N., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.25 · Cited by 811
Cliff´s Delta Calculator: A non-parametric effect size program for two groups of observations
Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2010) · Universitas Psychologica · doi:10.11144/javeriana.upsy10-2.cdcp · Cited by 442
Comprehensive benchmarking and ensemble approaches for metagenomic classifiers
McIntyre, A. B. R., Ounit, R., Afshinnekoo, E., Prill, R. J., Hénaff, E., Alexander, N., Minot, S. S., Danko, D., Foox, J., Ahsanuddin, S., et al. (2017) · Genome Biology · doi:10.1186/s13059-017-1299-7 · Cited by 342 · cited 1×
“…reads is not necessarily simple, with the performance of read classifiers depending in large part on the sequence database they query (such as, e.g., EMBL proGenomes [90] or NCBI RefSeq [91]), which can vary widely in scope, redundancy, and degree of curation (see performance evaluations [92, 93]).”
Background
Cited by
Diversity within Species: Interpreting Strains in Microbiomes
Van Rossum, T., Ferretti, P., Maistrenko, O. M., & Bork, P. (2020) · Nature Reviews Microbiology · doi:10.1038/s41579-020-0368-1 · Cited by 441
Dohlman, A. B., Arguijo Mendoza, D., Ding, S., Gao, M., Dressman, H., Iliev, I. D., Lipkin, S. M., & Shen, X. (2021) · Cell Host & Microbe · doi:10.1016/j.chom.2020.12.001 · Cited by 277
Carvalho, M. J., Sands, K., Thomson, K., Portal, E., Mathias, J., Milton, R., Gillespie, D., Dyer, C., Akpulu, C., Boostrom, I., et al. (2022) · Nature Microbiology · doi:10.1038/s41564-022-01184-y · Cited by 95
A Benchmark of Genetic Variant Calling Pipelines Using Metagenomic Short-Read Sequencing
Andreu-Sánchez, S., Chen, L., Wang, D., Augustijn, H. E., Zhernakova, A., & Fu, J. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.648229 · Cited by 27
(2024) · doi:10.1128/jb.00140-24 · Cited by 4
Metadata
- Authors
- Bush, S. J., Foster, D., Eyre, D. W., Clark, E. L., De Maio, N., Shaw, L. P., Stoesser, N., Peto, T. E. A., Crook, D. W., & Walker, A. S.
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-02-01
- Volume
- 9
- Issue
- 2
- ISSN
- 2047-217X (ISSN-L)
- OpenAlex
- W3005014603
- Cited by
- 140
- References
- 17
- FWCI
- 5.88
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.