The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
An evaluation of the accuracy and speed of metagenome analysis tools
Open accessAbstract
Metagenome studies are becoming increasingly widespread, yielding important insights into microbial communities covering diverse environments from terrestrial and aquatic ecosystems to human skin and gut. With the advent of high-throughput sequencing platforms, the use of large scale shotgun sequencing approaches is now commonplace. However, a thorough independent benchmark comparing state-of-the-art metagenome analysis tools is lacking. Here, we present a benchmark where the most widely used tools are tested on complex, realistic data sets. Our results clearly show that the most widely used tools are not necessarily the most accurate, that the most accurate tool is not necessarily the most time consuming and that there is a high degree of variability between available tools. These findings are important as the conclusions of any metagenomics study are affected by errors in the predicted community composition and functional capacity. Data sets and results are freely available from http://www.ucbioinformatics.org/metabenchmark.html
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669
Trimmomatic: a flexible trimmer for Illumina sequence data
Bolger, A. M., Lohse, M., & Usadel, B. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu170 · Cited by 65,548
Fast and accurate short read alignment with Burrows-Wheeler transform
Li, H., & Durbin, R. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp324 · Cited by 58,607
Fast gapped-read alignment with Bowtie 2
Langmead, B., & Salzberg, S. L. (2012) · Nature Methods · doi:10.1038/nmeth.1923 · Cited by 56,400
Gene Ontology: tool for the unification of biology
Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S., Eppig, J. T., et al. (2000) · Nature Genetics · doi:10.1038/75556 · Cited by 43,206
QIIME allows analysis of high-throughput community sequencing data
Caporaso, J. G., Kuczynski, J., Stombaugh, J., Bittinger, K., Bushman, F. D., Costello, E. K., Fierer, N., Peña, A. G., Goodrich, J. K., Gordon, J. I., et al. (2010) · Nature Methods · doi:10.1038/nmeth.f.303 · Cited by 34,887
vegan: Community Ecology Package
Oksanen, J., Simpson, G. L., Blanchet, F. G., Kindt, R., Legendre, P., Minchin, P. R., O'Hara, R. B., Solymos, P., Stevens, M. H. H., Szoecs, E., et al. (2001) · CRAN: Contributed Packages · doi:10.32614/cran.package.vegan · Cited by 22,615
Ultrafast and memory-efficient alignment of short DNA sequences to the human genome.
Langmead, B., Trapnell, C., Pop, M., & Salzberg, S. L. (2009) · Genome Biology · doi:10.1186/gb-2009-10-3-r25 · Cited by 21,849
Search and clustering orders of magnitude faster than BLAST
Edgar, R. C. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq461 · Cited by 20,826
Fast and sensitive protein alignment using DIAMOND
Buchfink, B., Xie, C., & Huson, D. H. (2015) · Nature Methods · doi:10.1038/nmeth.3176 · Cited by 14,201
The Pfam Protein Families Database
Bateman, A. (2002) · Nucleic Acids Research · doi:10.1093/nar/30.1.276 · Cited by 14,000
Greengenes, a Chimera-Checked 16S rRNA Gene Database and Workbench Compatible with ARB
DeSantis, T. Z., Hugenholtz, P., Larsen, N., Rojas, M., Brodie, E. L., Keller, K., Huber, T., Dalevi, D., Hu, P., & Andersen, G. L. (2006) · Applied and Environmental Microbiology · doi:10.1128/aem.03006-05 · Cited by 10,891
Predictive functional profiling of microbial communities using 16S rRNA marker gene sequences
Langille, M. G. I., Zaneveld, J., Caporaso, J. G., McDonald, D., Knights, D., Reyes, J. A., Clemente, J. C., Burkepile, D. E., Vega Thurber, R. L., Knight, R., et al. (2013) · Nature Biotechnology · doi:10.1038/nbt.2676 · Cited by 9,116
<tt>BLAT</tt> —The <tt>BLAST</tt> -Like Alignment Tool
Kent, W. J. (2002) · Genome Research · doi:10.1101/gr.229202 · Cited by 8,480
Genome sequencing in microfabricated high-density picolitre reactors
Margulies, M., Egholm, M., Altman, W. E., Attiya, S., Bader, J. S., Bemben, L. A., Berka, J., Braverman, M. S., Chen, Y.-J., Chen, Z., et al. (2005) · Nature · doi:10.1038/nature03959 · Cited by 7,693
Accelerated Profile HMM Searches
Eddy, S. R. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002195 · Cited by 6,975
Pruesse, E., Quast, C., Knittel, K., Fuchs, B. M., Ludwig, W., Peplies, J., & Glockner, F. O. (2007) · Nucleic Acids Research · doi:10.1093/nar/gkm864 · Cited by 6,668
Biopython: freely available python tools for computational molecular biology and bioinformatics
Cock, P. J. A., Antao, T., Chang, J. T., Chapman, B. A., Cox, C. J., Dalke, A., Friedberg, I., Hamelryck, T., Kauff, F., Wilczynski, B., et al. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp163 · Cited by 6,080
KEGG for integration and interpretation of large-scale molecular data sets
Kanehisa, M., Goto, S., Sato, Y., Furumichi, M., & Tanabe, M. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr988 · Cited by 4,848
Kraken: ultrafast metagenomic sequence classification using exact alignments
Wood, D. E., & Salzberg, S. L. (2014) · Genome Biology · doi:10.1186/gb-2014-15-3-r46 · Cited by 4,521
The Pfam protein families database
Punta, M., Coggill, P. C., Eberhardt, R. Y., Mistry, J., Tate, J., Boursnell, C., Pang, N., Forslund, K., Ceric, G., Clements, J., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr1065 · Cited by 3,719
Meyer, F., Paarmann, D., D'Souza, M., Olson, R., Glass, E., Kubal, M., Paczian, T., Rodriguez, A., Stevens, R., Wilke, A., et al. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-386 · Cited by 3,565
MEGAN analysis of metagenomic data
Huson, D. H., Auch, A. F., Qi, J., & Schuster, S. C. (2007) · Genome Research · doi:10.1101/gr.5969107 · Cited by 3,312
Antibiotic resistance is ancient
D’Costa, V. M., King, C. E., Kalan, L., Morar, M., Sung, W. W. L., Schwarz, C., Froese, D., Zazula, G., Calmels, F., Debruyne, R., et al. (2011) · Nature · doi:10.1038/nature10388 · Cited by 2,720
Gut–brain axis: How the microbiome influences anxiety and depression
Foster, J. A., & McVey Neufeld, K.-A. (2013) · Trends in Neurosciences · doi:10.1016/j.tins.2013.01.005 · Cited by 2,460
The Subsystems Approach to Genome Annotation and its Use in the Project to Annotate 1000 Genomes
Overbeek, R. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki866 · Cited by 2,012
Metagenomic microbial community profiling using unique clade-specific marker genes
Segata, N., Waldron, L., Ballarini, A., Narasimhan, V., Jousson, O., & Huttenhower, C. (2012) · Nature Methods · doi:10.1038/nmeth.2066 · Cited by 1,776
Cross-biome metagenomic analyses of soil microbial communities and their functional attributes
Fierer, N., Leff, J. W., Adams, B. J., Nielsen, U. N., Bates, S. T., Lauber, C. L., Owens, S., Gilbert, J. A., Wall, D. H., & Caporaso, J. G. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1215210110 · Cited by 1,643
ART: a next-generation sequencing read simulator
Huang, W., Li, L., Myers, J. R., & Marth, G. T. (2012) · Bioinformatics · doi:10.1093/bioinformatics/btr708 · Cited by 1,596
New and continuing developments at PROSITE
Sigrist, C. J. A., de Castro, E., Cerutti, L., Cuche, B. A., Hulo, N., Bridge, A., Bougueleret, L., & Xenarios, I. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks1067 · Cited by 1,562
Integrative analysis of environmental sequences using MEGAN4
Huson, D. H., Mitra, S., Ruscheweyh, H.-J., Weber, N., & Schuster, S. C. (2011) · Genome Research · doi:10.1101/gr.120618.111 · Cited by 1,541
Cole, J. R. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg039 · Cited by 1,429
Accurate Multiplex Polony Sequencing of an Evolved Bacterial Genome
Shendure, J., Porreca, G. J., Reppas, N. B., Lin, X., McCutcheon, J. P., Rosenbaum, A. M., Wang, M. D., Zhang, K., Mitra, R. D., & Church, G. M. (2005) · Science · doi:10.1126/science.1117389 · Cited by 1,402
Rfam 12.0: updates to the RNA families database
Nawrocki, E. P., Burge, S. W., Bateman, A., Daub, J., Eberhardt, R. Y., Eddy, S. R., Floden, E. W., Gardner, P. P., Jones, T. A., Tate, J., et al. (2015) · Nucleic Acids Research · doi:10.1093/nar/gku1063 · Cited by 1,071
Self-citationInterPro in 2011: new developments in the family and domain prediction database
Hunter, S., Jones, P., Mitchell, A., Apweiler, R., Attwood, T. K., Bateman, A., Bernard, T., Binns, D., Bork, P., Burge, S., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr948 · Cited by 1,035
FragGeneScan: predicting genes in short and error-prone reads
Rho, M., Tang, H., & Ye, Y. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq747 · Cited by 840
Relating the metatranscriptome and metagenome of the human gut
Franzosa, E. A., Morgan, X. C., Segata, N., Waldron, L., Reyes, J., Earl, A. M., Giannoukos, G., Boylan, M. R., Ciulla, D., Gevers, D., et al. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1319284111 · Cited by 704
Inhalable Microorganisms in Beijing’s PM2.5 and PM10 Pollutants during a Severe Smog Event
Cao, C., Jiang, W., Wang, B., Fang, J., Lang, J., Tian, G., Jiang, J., & Zhu, T. F. (2014) · Environmental Science & Technology · doi:10.1021/es4048472 · Cited by 669
Ounit, R., Wanamaker, S., Close, T. J., & Lonardi, S. (2015) · BMC Genomics · doi:10.1186/s12864-015-1419-2 · Cited by 632
Nicol, J. W., Helt, G. A., Blanchard, S. G., Raja, A., & Loraine, A. E. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp472 · Cited by 624
Adler, C. J., Dobney, K., Weyrich, L. S., Kaidonis, J., Walker, A. W., Haak, W., Bradshaw, C. J. A., Townsend, G., Sołtysiak, A., Alt, K. W., et al. (2013) · Nature Genetics · doi:10.1038/ng.2536 · Cited by 621
Metagenomic species profiling using universal phylogenetic marker genes
Sunagawa, S., Mende, D. R., Zeller, G., Izquierdo-Carrasco, F., Berger, S. A., Kultima, J. R., Coelho, L. P., Arumugam, M., Tap, J., Nielsen, H. B., et al. (2013) · Nature Methods · doi:10.1038/nmeth.2693 · Cited by 516
A simple, fast, and accurate method of phylogenomic inference
Wu, M., & Eisen, J. A. (2008) · Genome Biology · doi:10.1186/gb-2008-9-10-r151 · Cited by 507
Flagella and bacterial pathogenicity
Duan, Q., Zhou, M., Zhu, L., & Zhu, G. (2013) · Journal of Basic Microbiology · doi:10.1002/jobm.201100335 · Cited by 488
RefSeq microbial genomes database: new representation and annotation strategy
Tatusova, T., Ciufo, S., Fedorov, B., O’Neill, K., & Tolstoy, I. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1274 · Cited by 483
The microbiome and rheumatoid arthritis
Scher, J. U., & Abramson, S. B. (2011) · Nature Reviews Rheumatology · doi:10.1038/nrrheum.2011.121 · Cited by 474
Ancient and modern environmental DNA
Pedersen, M. W., Overballe-Petersen, S., Ermini, L., Sarkissian, C. D., Haile, J., Hellstrom, M., Spens, J., Thomsen, P. F., Bohmann, K., Cappellini, E., et al. (2015) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0383 · Cited by 329
The Human Gut Microbiome and Body Metabolism: Implications for Obesity and Diabetes
Devaraj, S., Hemarajata, P., & Versalovic, J. (2013) · Clinical Chemistry · doi:10.1373/clinchem.2012.187617 · Cited by 321
Wilke, A., Harrison, T., Wilkening, J., Field, D., Glass, E. M., Kyrpides, N., Mavrommatis, K., & Meyer, F. (2012) · BMC Bioinformatics · doi:10.1186/1471-2105-13-141 · Cited by 314
Rose: generating sequence families.
Stoye, J., Evers, D., & Meyer, F. (1998) · Bioinformatics · doi:10.1093/bioinformatics/14.2.157 · Cited by 265
Parameters for accurate genome alignment
Frith, M. C., Hamada, M., & Horton, P. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-80 · Cited by 229
Insights from Characterizing Extinct Human Gut Microbiomes
Tito, R. Y., Knights, D., Metcalf, J., Obregon-Tito, A. J., Cleeland, L., Najar, F., Roe, B., Reinhard, K., Sobolik, K., Belknap, S., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051146 · Cited by 216
Shakya, M., Quince, C., Campbell, J. H., Yang, Z. K., Schadt, C. W., & Podar, M. (2013) · Environmental Microbiology · doi:10.1111/1462-2920.12086 · Cited by 214
Scalable metagenomic taxonomy classification using a reference genome database
Ames, S. K., Hysom, D. A., Gardner, S. N., Lloyd, G. S., Gokhale, M. B., & Allen, J. E. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt389 · Cited by 198
Accurate read-based metagenome characterization using a hierarchical suite of unique signatures
Freitas, T. A. K., Li, P.-E., Scholz, M. B., & Chain, P. S. G. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv180 · Cited by 179
Dröge, J., Gregor, I., & McHardy, A. C. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu745 · Cited by 159
Hospital disinfectants and spore formation by Clostridium difficile
Wilcox, M. H., & Fawley, W. N. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02819-1 · Cited by 157
EBI metagenomics—a new resource for the analysis and archiving of metagenomic data
Hunter, S., Corbett, M., Denise, H., Fraser, M., Gonzalez-Beltran, A. N., Hunter, C., JONES, P. M., Leinonen, R., McAnulla, C., Maguire, E., et al. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt961 · Cited by 156
Fierer, N., Barberán, A., & Laughlin, D. C. (2014) · Frontiers in Microbiology · doi:10.3389/fmicb.2014.00614 · Cited by 139
Lees, J., Yeats, C., Perkins, J. R., Sillitoe, I., Rentzsch, R., Dessailly, B. H., & Orengo, C. A. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr1181 · Cited by 120
Booijink, C. C., Boekhorst, J., Zoetendal, E. G., Smidt, H., Kleerebezem, M., & de Vos, W. M. (2010) · Applied and Environmental Microbiology · doi:10.1128/aem.00502-10 · Cited by 90
MetaPhyler: Taxonomic profiling for metagenomic sequences
Liu, B., Gibbons, T., Ghodsi, M., & Pop, M. (2010) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2010.5706544 · Cited by 76
Lee, J., Yi, H., & Chun, J. (2011) · The Journal of Microbiology · doi:10.1007/s12275-011-1213-z · Cited by 51
Genometa - A Fast and Accurate Classifier for Short Metagenomic Shotgun Reads
Davenport, C., Neugebauer, J., Beckmann, N., Friedrich, B., Kameri, B., Kokott, S., Paetow, M., Siekmann, B., Wieding-Drewes, M., Wienhöfer, M., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041224 · Cited by 43
Gama, J. A., Abby, S. S., Vieira‐Silva, S., Dionísio, F., & Rocha, E. P. C. (2012) · PLoS Pathogens · doi:10.1371/journal.ppat.1002503 · Cited by 43
Metatranscriptome profiling of a harmful algal bloom
Cooper, E. D., Bentlage, B., Gibbons, T., Bachvaroff, T. R., & Delwiche, C. F. (2014) · Harmful Algae · doi:10.1016/j.hal.2014.04.016 · Cited by 41
Cited by
Bracken: estimating species abundance in metagenomics data
(2016) · doi:10.7717/peerj-cs.104 · Cited by 2,333
Fast and sensitive taxonomic classification for metagenomics with Kaiju
Menzel, P., Ng, K. L., & Krogh, A. (2016) · Nature Communications · doi:10.1038/ncomms11257 · Cited by 2,118
Huson, D. H., Beier, S., Flade, I., Górska, A., El-Hadidi, M., Mitra, S., Ruscheweyh, H.-J., & Tappu, R. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004957 · Cited by 1,761
Centrifuge: rapid and sensitive classification of metagenomic sequences
Kim, D., Song, L., Breitwieser, F. P., & Salzberg, S. L. (2016) · Genome Research · doi:10.1101/gr.210641.116 · Cited by 1,438
Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software
Sczyrba, A., Hofmann, P., Belmann, P., Koslicki, D., Janssen, S., Dröge, J., Gregor, I., Majda, S., Fiedler, J., Dahms, E., et al. (2017) · Nature Methods · doi:10.1038/nmeth.4458 · Cited by 870
Metadata
- Authors
- Lindgreen, S., Adair, K. L., & Gardner, P. P.
- Venue
- Scientific Reports
- Published
- 2016-01-18
- Volume
- 6
- Issue
- 1
- Pages
- 19233-19233
- ISSN
- 2045-2322 (ISSN-L)
- OpenAlex
- W2951176108
- Cited by
- 331
- References
- 66
- FWCI
- 25.58
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
