The DOI matches its Crossref record; 79 of the 80 references it cites were found in a registry or an index, and 1 are not checked yet.
Comprehensive benchmarking and ensemble approaches for metagenomic classifiers
Open accessPublished version of a preprint
Abstract
BACKGROUND: One of the main challenges in metagenomics is the identification of microorganisms in clinical and environmental samples. While an extensive and heterogeneous set of computational tools is available to classify microorganisms using whole-genome shotgun sequencing data, comprehensive comparisons of these methods are limited. RESULTS: In this study, we use the largest-to-date set of laboratory-generated and simulated controls across 846 species to evaluate the performance of 11 metagenomic classifiers. Tools were characterized on the basis of their ability to identify taxa at the genus, species, and strain levels, quantify relative abundances of taxa, and classify individual reads to the species level. Strikingly, the number of species identified by the 11 tools can differ by over three orders of magnitude on the same datasets. Various strategies can ameliorate taxonomic misclassification, including abundance filtering, ensemble approaches, and tool intersection. Nevertheless, these strategies were often insufficient to completely eliminate false positives from environmental samples, which are especially important where they concern medically relevant species. Overall, pairing tools with different classification strategies (k-mer, alignment, marker) can combine their respective advantages. CONCLUSIONS: This study provides positive and negative controls, titrated standards, and a guide for selecting tools for metagenomic analyses by comparing ranges of precision, accuracy, and recall. We show that proper experimental design and analysis parameters can reduce false positives, provide greater resolution of species in complex metagenomic samples, and improve the interpretation of results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These three profiling tools have consistently been shown to outperform other methods across multiple evaluations (McIntyre et al., 2017; Meyer et al., 2019; Milanese et al., 2019; Sczyrba et al., 2017; Truong et al., 2015; Ye et al., 2019).”
From Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3 (2021) · cites this paper as McIntyre et al., 2017
Experimental — accuracy not yet measuredIn the citing paper
“Consistently, more bacterial taxa were identified per sample with Kraken2 than Metaphlan2, a result of the updated reference database and higher sensitivity of this k-mer–based approach (Fig. S2, A–C; McIntyre et al., 2017).”
From Gut microbiome stability and dynamics in healthy donors and patients with non-gastrointestinal cancers (2021) · cites this paper as McIntyre et al., 2017
In the citing paper
“Most available metagenomic reference materials are highly customized to specific applications (for example, ZymoBIOMICS Microbial Community Standard for microbiome analyses and bacterial and fungal metagenomics105) and/or focused on a more limited spectrum of organisms (for example, the National Institute of Standards and Technology (NIST) reference materials for mixed microbial DNA detection, wh”
From Clinical metagenomics (2019) · cites this paper as [105]
In the citing paper
“The use of standardized reference materials and NGS data sets is also helpful in comparative evaluation of different bioinformatics pipelines105.”
From Clinical metagenomics (2019) · cites this paper as [105]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Ashrafi, I., Kohram, H., Naijian, H., Bahreini, M., & Mirzakhani, H. (2012) · BMC Research Notes · doi:10.1186/1756-0500-5-319 · Cited by 100
RetractedGapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302 · cited 1×
“We compared the characteristics and parameters of a set of 11 metagenomic tools [14, 33 – 44] (Additional file 1: Table S1) representing a variety of classification approaches ( k -mer composition, alignment, marker).”
MethodFast and sensitive protein alignment using DIAMOND
Buchfink, B., Xie, C., & Huson, D. H. (2015) · Nature Methods · doi:10.1038/nmeth.3176 · Cited by 14,201
16S ribosomal DNA amplification for phylogenetic study
Weisburg, W. G., Barns, S. M., Pelletier, D. A., & Lane, D. J. (1991) · Journal of Bacteriology · doi:10.1128/jb.173.2.697-703.1991 · Cited by 11,934 · cited 1×
“Highly conserved regions within these genes permit the use of common primers for sequencing [7].”
BackgroundA human gut microbial gene catalogue established by metagenomic sequencing
MetaHIT Consortium, Qin, J., Li, R., Raes, J., Arumugam, M., Burgdorf, K. S., Manichanh, C., Nielsen, T., Pons, N., Levenez, F., et al. (2010) · Nature · doi:10.1038/nature08821 · Cited by 11,624 · cited 1×
“For instance, highly complex microbial communities found in the adult human gut and in soil contain species numbering in the hundreds and tens of thousands, respectively [61, 62].”
BackgroundKraken: ultrafast metagenomic sequence classification using exact alignments
Wood, D. E., & Salzberg, S. L. (2014) · Genome Biology · doi:10.1186/gb-2014-15-3-r46 · Cited by 4,521
Reagent and laboratory contamination can critically impact sequence-based microbiome analyses
Salter, S. J., Cox, M. J., Turek, E. M., Calus, S. T., Cookson, W. O., Moffatt, M. F., Turner, P., Parkhill, J., Loman, N. J., & Walker, A. W. (2014) · BMC Biology · doi:10.1186/s12915-014-0087-z · Cited by 3,463 · cited 2×
“Previous studies have also detected contamination with both [46].”
BackgroundResultMEGAN analysis of metagenomic data
Huson, D. H., Auch, A. F., Qi, J., & Schuster, S. C. (2007) · Genome Research · doi:10.1101/gr.5969107 · Cited by 3,312 · cited 1×
“Briefly, six data-sets were created using the ART simulator with default error and quality base profiles [66] to simulate 100-bp Illumina reads from sets of reference sequences at a coverage of 30X and efficiently post-processed to remove ambigu-ously mapped read at the species levels [36].”
MethodDysfunction of the intestinal microbiome in inflammatory bowel disease and treatment
Morgan, X. C., Tickle, T. L., Sokol, H., Gevers, D., Devaney, K. L., Ward, D. V., Reyes, J. A., Shah, S. A., LeLeiko, N., Snapper, S. B., et al. (2012) · Genome Biology · doi:10.1186/gb-2012-13-9-r79 · Cited by 2,713 · cited 1×
“Sequencing has helped researchers identify microorganisms with roles in such diverse areas as human health [1], the color of lakes [2], and climate [3, 4].”
BackgroundBracken: estimating species abundance in metagenomics data
(2016) · doi:10.7717/peerj-cs.104 · Cited by 2,333 · cited 1×
“A similar study by Peabody, et al. [21] evaluated algorithms to the species level but included only two datasets representing 11 species, without taking into account the evolution of the taxonomy of those species [31].”
Result
Research works printed but not linked: 1
- Minot SS, Greenfield N, Afshinnekoo E, Mason CE. Detection of Bacillus anthracis using a targeted gene panel. 2015. https://science.onecodex.com/bacillus-anthracis-panel/(accessed 29 Dec 2016).
Cited by
Beghini, F., McIver, L. J., Blanco-Míguez, A., Dubois, L., Asnicar, F., Maharjan, S., Mailyan, A., Manghi, P., Scholz, M., Thomas, A. M., et al. (2021) · eLife · doi:10.7554/elife.65088 · Cited by 2,303
Best practices for analysing microbiomes
Knight, R., Vrbanac, A., Taylor, B. C., Aksenov, A., Callewaert, C., Debelius, J., Gonzalez, A., Kosciolek, T., McCall, L.-I., McDonald, D., et al. (2018) · Nature Reviews Microbiology · doi:10.1038/s41579-018-0029-9 · Cited by 2,030
Chiu, C. Y., & Miller, S. A. (2019) · Nature Reviews Genetics · doi:10.1038/s41576-019-0113-7 · Cited by 1,620
Rapid pathogen detection by metagenomic next-generation sequencing of infected body fluids
Gu, W., Deng, X., Lee, M., Sucu, Y. D., Arevalo, S., Stryke, D., Federman, S., Gopez, A., Reyes, K., Zorn, K., et al. (2021) · Nature Medicine · doi:10.1038/s41591-020-1105-z · Cited by 689
Benchmarking Metagenomics Tools for Taxonomic Classification
Ye, S. H., Siddle, K. J., Park, D. J., & Sabeti, P. C. (2019) · Cell · doi:10.1016/j.cell.2019.07.010 · Cited by 593
“Thinking more broadly, one proposed strategy to address the individual shortcomings of different classifiers is to use an ensemble of classifiers, or a pipeline of different classifiers (McIntyre et al., 2017; Piro et al., 2017).”
BackgroundResultInfluential
Metadata
- Authors
- McIntyre, A. B. R., Ounit, R., Afshinnekoo, E., Prill, R. J., Hénaff, E., Alexander, N., Minot, S. S., Danko, D., Foox, J., Ahsanuddin, S., Tighe, S., Hasan, N. A., Subramanian, P., Moffat, K., Levy, S., Lonardi, S., Greenfield, N., Colwell, R. R., Rosen, G. L., & Mason, C. E.
- Affiliations
- Weill Cornell Medicine (US)
- University of California, Riverside (US)
- New York Medical College (US)
- IBM (United States) (US)
- University of Vermont (US)
- University of Maryland, College Park (US)
- HudsonAlpha Institute for Biotechnology (US)
- Johns Hopkins University (US)
- Drexel University (US)
- Corresponding author
- Rosen, Gail L.
- Mason, Christopher E.
- ORCID
- Venue
- Genome Biology
- Published
- 2017-09-21
- Volume
- 18
- Issue
- 1
- Pages
- 182-182
- ISSN
- 1465-6906 (ISSN-L)
- 1474-760X
- PMC
- PMC5609029
- OpenAlex
- W2952152744
- Cited by
- 342
- References
- 78
- Citations by year
- 2026: 31 · 2025: 18 · 2024: 43 · 2023: 30 · 2022: 49 · 2021: 51 · 2020: 58 · 2019: 37 · 2018: 21 · 2017: 4
- FWCI
- 13.41
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-04-05)
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Genome biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- Bill and Melinda Gates Foundation (OPP1151054)
- National Aeronautics and Space Administration (15-15Omni2-0063, NNX14AH50G)
- National Institutes of Health (R01AI125416)
- Alfred P. Sloan Foundation (G-2015-13964)
- Starr Foundation (I9-A9-071)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Physical Sciences › Environmental Science › Ecology › Environmental DNA in Biodiversity Studies
- Health Sciences › Medicine › Microbiology (medical) › Bacterial Identification and Susceptibility Testing
- Keywords
- metagenomic classification, benchmarking, whole genome shotgun sequencing, taxonomic classification, species-level classification, ensemble methods, false positives
- MeSH terms
- Humans, Phylogeny, Software, Sequence Analysis, DNA, Benchmarking, Contig Mapping, Metagenome, DNA Barcoding, Taxonomic, Microbiota
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comprehensive benchmarking and ensemble approaches for metagenomic classifiers. https://app.cochat.ai/papers/10.1186/s13059-017-1299-7
- The paper
- McIntyre, A. B. R., Ounit, R., Afshinnekoo, E., Prill, R. J., Hénaff, E., Alexander, N., Minot, S. S., Danko, D., Foox, J., Ahsanuddin, S., Tighe, S., Hasan, N. A., Subramanian, P., Moffat, K., Levy, S., Lonardi, S., Greenfield, N., Colwell, R. R., Rosen, G. L., & Mason, C. E. (2017). Comprehensive benchmarking and ensemble approaches for metagenomic classifiers. Genome Biology. https://doi.org/10.1186/s13059-017-1299-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.