The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Identifying and Overcoming Threats to Reproducibility, Replicability, Robustness, and Generalizability in Microbiome Research
Open accessAbstract
The “reproducibility crisis” in science affects microbiology as much as any other area of inquiry, and microbiologists have long struggled to make their research reproducible. We need to respect that ensuring that our methods and results are sufficiently transparent is difficult. This difficulty is compounded in interdisciplinary fields such as microbiome research. There are many reasons why a researcher is unable to reproduce a previous result, and even if a result is reproducible, it may not be correct. Furthermore, failures to reproduce previous results have much to teach us about the scientific process and microbial life itself. This Perspective delineates a framework for identifying and overcoming threats to reproducibility, replicability, robustness, and generalizability of microbiome research. Instead of seeing signs of a crisis in others’ work, we need to appreciate the technical and social difficulties that limit reproducibility in the work of others as well as our own.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schloss, P. D., Westcott, S. L., Ryabin, T., Hall, J. R., Hartmann, M., Hollister, E. B., Lesniewski, R. A., Oakley, B. B., Parks, D. H., Robinson, C. J., et al. (2009) · Applied and Environmental Microbiology · doi:10.1128/aem.01541-09 · Cited by 19,549
Self-citationAn obesity-associated gut microbiome with increased capacity for energy harvest
Turnbaugh, P. J., Ley, R. E., Mahowald, M. A., Magrini, V., Mardis, E. R., & Gordon, J. I. (2006) · Nature · doi:10.1038/nature05414 · Cited by 12,444
Structure, function and diversity of the healthy human microbiome
The Human Microbiome Project Consortium (2012) · Nature · doi:10.1038/nature11234 · Cited by 11,726
Greengenes, a Chimera-Checked 16S rRNA Gene Database and Workbench Compatible with ARB
DeSantis, T. Z., Hugenholtz, P., Larsen, N., Rojas, M., Brodie, E. L., Keller, K., Huber, T., Dalevi, D., Hu, P., & Andersen, G. L. (2006) · Applied and Environmental Microbiology · doi:10.1128/aem.03006-05 · Cited by 10,891
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Human gut microbes associated with obesity
Ley, R. E., Turnbaugh, P. J., Klein, S., & Gordon, J. I. (2006) · Nature · doi:10.1038/4441022a · Cited by 8,785
A core gut microbiome in obese and lean twins
Turnbaugh, P. J., Hamady, M., Yatsunenko, T., Cantarel, B. L., Duncan, A., Ley, R. E., Sogin, M. L., Jones, W. J., Roe, B. A., Affourtit, J. P., et al. (2009) · Nature · doi:10.1038/nature07540 · Cited by 7,823
Obesity alters gut microbial ecology
Ley, R. E., Bäckhed, F., Turnbaugh, P., Lozupone, C. A., Knight, R. D., & Gordon, J. I. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0504978102 · Cited by 6,176
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“In other cases, the original study may have been underpowered, rendering it susceptible to an inflated risk of type I errors (38).”
BackgroundInduction of Intestinal Th17 Cells by Segmented Filamentous Bacteria
Ivanov, I. I., Atarashi, K., Manel, N., Brodie, E. L., Shima, T., Karaoz, U., Wei, D., Goldfarb, K. C., Santee, C. A., Lynch, S. V., et al. (2009) · Cell · doi:10.1016/j.cell.2009.09.033 · Cited by 4,554
Ribosomal Database Project: data and tools for high throughput rRNA analysis
Cole, J. R., Wang, Q., Fish, J. A., Chai, B., McGarrell, D. M., Sun, Y., Brown, C. T., Porras-Alfaro, A., Kuske, C. R., & Tiedje, J. M. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1244 · Cited by 4,367
The SILVA and “All-species Living Tree Project (LTP)” taxonomic frameworks
Yilmaz, P., Parfrey, L. W., Yarza, P., Gerken, J., Pruesse, E., Quast, C., Schweer, T., Peplies, J., Ludwig, W., & Glöckner, F. O. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1209 · Cited by 3,952
Reagent and laboratory contamination can critically impact sequence-based microbiome analyses
Salter, S. J., Cox, M. J., Turek, E. M., Calus, S. T., Cookson, W. O., Moffatt, M. F., Turner, P., Parkhill, J., Loman, N. J., & Walker, A. W. (2014) · BMC Biology · doi:10.1186/s12915-014-0087-z · Cited by 3,463
Jupyter Notebooks – a publishing format for reproducible computational workflows
Thomas, K., Benjamin, R.-K., Fernando, P., Brian, G., Matthias, B., Jonathan, F., Kyle, K., Jessica, H., Jason, G., Sylvain, C., et al. (2016) · Positioning and Power in Academic Publishing: Players, Agents and Agendas · doi:10.3233/978-1-61499-649-1-87 · Cited by 3,063
Genomic responses in mouse models poorly mimic human inflammatory diseases
Seok, J., Warren, H. S., Cuenca, A. G., Mindrinos, M. N., Baker, H. V., Xu, W., Richards, D. R., McDonald-Smith, G. P., Gao, H., Hennessy, L., et al. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1222878110 · Cited by 3,005
Tackling the widespread and critical impact of batch effects in high-throughput data
Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Johnson, W. E., Geman, D., Baggerly, K., & Irizarry, R. A. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2825 · Cited by 2,196
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Population-based metagenomics analysis reveals markers for gut microbiome composition and diversity
Zhernakova, A., Kurilshikov, A., Bonder, M. J., Tigchelaar, E. F., Schirmer, M., Vatanen, T., Mujagic, Z., Vila, A. V., Falony, G., Vieira-Silva, S., et al. (2016) · Science · doi:10.1126/science.aad3369 · Cited by 1,914
Ivanov, I. I., Frutos, R. D. L., Manel, N., Yoshinaga, K., Rifkin, D. B., Sartor, R. B., Finlay, B. B., & Littman, D. R. (2008) · Cell Host & Microbe · doi:10.1016/j.chom.2008.09.009 · Cited by 1,770
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 1×
“In reality, questions change and scientists can fall into the traps of the “Garden of Many Forking Paths,” where they go looking for a desired result (32), or “P-hacking,” where large numbers of statistical hypothesis tests are attempted without adequately correcting for performing multiple tests (33).”
BackgroundHow informative is the mouse for human gut microbiota research?
Nguyen, T. L. A., Vieira-Silva, S., Liston, A., & Raes, J. (2015) · Disease Models & Mechanisms · doi:10.1242/dmm.017400 · Cited by 1,334 · cited 1×
“However, it is not always trivial to generalize that knowledge to related species and strains or from in vitro to in vivo conditions and on to human subjects (62, 63).”
BackgroundWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Perez-Muñoz, M. E., Arrieta, M.-C., Ramer-Tait, A. E., & Walter, J. (2017) · Microbiome · doi:10.1186/s40168-017-0268-4 · Cited by 1,096
Genetic determinants of the gut microbiome in UK twins
Goodrich, J. K., Davenport, E. R., Beaumont, M., Jackson, M. A., Knight, R., Ober, C., Spector, T. D., Bell, J. T., Clark, A. G., & Ley, R. E. (2016) · Cell Host & Microbe · doi:10.1016/j.chom.2016.04.017 · Cited by 1,048
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Dynamics and associations of microbial community types across the human body
Ding, T., & Schloss, P. D. (2014) · Nature · doi:10.1038/nature13178 · Cited by 885
Self-citationTen Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880
Meta‐analyses of human gut microbes associated with obesity and IBD
Walters, W. A., Xu, Z., & Knight, R. (2014) · FEBS Letters · doi:10.1016/j.febslet.2014.09.039 · Cited by 836
Comparison of the Respiratory Microbiome in Healthy Nonsmokers and Smokers
Morris, A., Beck, J. M., Schloss, P. D., Campbell, T. B., Crothers, K., Curtis, J. L., Flores, S. C., Fontenot, A. P., Ghedin, E., Huang, L., et al. (2013) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201210-1913oc · Cited by 822
Self-citationDynamic Documents with R and knitr
Xie, Y. (2016) · Chapman & Hall/CRC eBooks · doi:10.1201/b15166 · Cited by 746
Robust research needs many lines of evidence
Munafò, M. R., & Davey Smith, G. (2018) · Nature · doi:10.1038/d41586-018-01023-3 · Cited by 666
Looking for a Signal in the Noise: Revisiting Obesity and the Microbiome
Sze, M. A., & Schloss, P. D. (2016) · mBio · doi:10.1128/mbio.01018-16 · Cited by 600 · cited 4×
“Several research groups, including mine (11), have attempted to validate the result that obese individuals were more likely to have lower bacterial diversity and relative abundances of Bacteroidetes (12, 13).”
BackgroundMethodResultInfluentialSelf-citationOptimizing methods and dodging pitfalls in microbiome research
Kim, D., Hofstaedter, C. E., Zhao, C., Mattei, L., Tanes, C., Clarke, E., Lauder, A., Sherrill-Mix, S., Chehoud, C., Kelsen, J., et al. (2017) · Microbiome · doi:10.1186/s40168-017-0267-5 · Cited by 541
Lauder, A. P., Roche, A. M., Sherrill-Mix, S., Bailey, A., Laughlin, A. L., Bittinger, K., Leite, R., Elovitz, M. A., Parry, S., & Bushman, F. D. (2016) · Microbiome · doi:10.1186/s40168-016-0172-3 · Cited by 539
Heterogeneity of the gut microbiome in mice: guidelines for optimizing experimental design
Laukens, D., Brinkman, B. M., Raes, J., De Vos, M., & Vandenabeele, P. (2016) · FEMS Microbiology Reviews · doi:10.1093/femsre/fuv036 · Cited by 379
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
A Taxonomic Signature of Obesity in the Microbiome? Getting to the Guts of the Matter
Finucane, M. M., Sharpton, T. J., Laurent, T. J., & Pollard, K. S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084689 · Cited by 321 · cited 3×
“Several research groups, including mine (11), have attempted to validate the result that obese individuals were more likely to have lower bacterial diversity and relative abundances of Bacteroidetes (12, 13).”
BackgroundThe unseen world: reflections on Leeuwenhoek (1677) ‘Concerning little animals’
Lane, N. (2015) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2014.0344 · Cited by 313
An open investigation of the reproducibility of cancer biology research
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J., & Nosek, B. A. (2014) · eLife · doi:10.7554/elife.04333 · Cited by 312
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · cited 1×
“Solutions to these problems include preregistering data analysis plans (34–36), justifying sample sizes based on power calculations (10, 11, 37), and using Bayesian frameworks that allow prior knowledge of the system to influence the interpretation of new results (39, 40).”
BackgroundPatil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243
Ten Simple Rules for Taking Advantage of Git and GitHub
Perez-Riverol, Y., Gatto, L., Wang, R., Sachsenberg, T., Uszkoreit, J., Leprevost, F. D. V., Fufezan, C., Ternent, T., Eglen, S. J., Katz, D. S., et al. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004947 · Cited by 230
Analysis of the Gut Microbiota in the Old Order Amish and Its Relation to the Metabolic Syndrome
Zupancic, M. L., Cantarel, B. L., Liu, Z., Drabek, E. F., Ryan, K. A., Cirimotich, S., Jones, C., Knight, R., Walters, W. A., Knights, D., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043052 · Cited by 214
Common Workflow Language, draft 3
Amstutz, P., Andeer, R., Chapman, B., Chilton, J., Crusoe, M. R., Guimerà, R. V., Hernandez, G. C., Ivkovic, S., Kartashov, A., Kern, J., et al. (2016) · eScholarship (California Digital Library) · doi:10.6084/m9.figshare.3115156 · Cited by 204
Reproducible research can still be wrong: Adopting a prevention approach
Leek, J. T., & Peng, R. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1421412111 · Cited by 193
Experimental design and quantitative analysis of microbial community multiomics
Mallick, H., Ma, S., Franzosa, E. A., Vatanen, T., Morgan, X. C., & Huttenhower, C. (2017) · Genome Biology · doi:10.1186/s13059-017-1359-z · Cited by 192
Scholarly Context Not Found: One in Five Articles Suffers from Reference Rot
Klein, M., Van de Sompel, H., Sanderson, R., Shankar, H., Balakireva, L., Zhou, K., & Tobin, R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0115253 · Cited by 184
Software Carpentry: lessons learned
Wilson, G. (2016) · F1000Research · doi:10.12688/f1000research.3-62.v2 · Cited by 174
The ghosts of HeLa: How cell line misidentification contaminates the scientific literature
Horbach, S. P. J. M., & Halffman, W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186281 · Cited by 167 · cited 1×
“Perhaps the best-known example is the discovery that HeLa cells contaminate many other cell lines, especially those in the same laboratory (48, 49).”
BackgroundJenior, M. L., Leslie, J. L., Young, V. B., & Schloss, P. D. (2017) · mSystems · doi:10.1128/msystems.00063-17 · Cited by 141
Self-citationCasadevall, A., & Fang, F. C. (2010) · Infection and Immunity · doi:10.1128/iai.00908-10 · Cited by 133
Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome
Garijo, D., Kinnings, S., Xie, L., Xie, L., Zhang, Y., Bourne, P. E., & Gil, Y. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080278 · Cited by 132
Ten Simple Rules for Digital Data Storage
Hart, E., Barmby, P., LeBauer, D. S., Michonneau, F., Mount, S., Mulrooney, P., Poisot, T., Woo, K., Zimmerman, N. B., & Hollister, J. W. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005097 · Cited by 124
Ten simple rules for making research software more robust
Taschuk, M. L., & Wilson, G. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005412 · Cited by 111
The Mycobacterium tuberculosis Drugome and Its Polypharmacological Implications
Kinnings, S., Xie, L., Fung, K. H., Jackson, R. M., Xie, L., & Bourne, P. E. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000976 · Cited by 109
Investigation of Cross-Contamination and Misidentification of 278 Widely Used Tumor Cell Lines
Huang, Y., Liu, Y., Zheng, C., & Shen, C. (2017) · PLOS ONE · doi:10.1371/journal.pone.0170384 · Cited by 105
A Quick Guide to Organizing Computational Biology Projects
Noble, W. S. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000424 · Cited by 97
Dynamic Documents with R and knitr, Second Edition
Xie, Y. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315382487 · Cited by 95
Multi-omics Analysis of Periodontal Pocket Microbial Communities Pre- and Posttreatment
Califf, K. J., Schwarzberg-Lipson, K., Garg, N., Gibbons, S. M., Caporaso, J. G., Slots, J., Cohen, C., Dorrestein, P. C., & Kelley, S. T. (2017) · mSystems · doi:10.1128/msystems.00016-17 · Cited by 66
“Available upon request”: not good enough for microbiome data!
Langille, M. G. I., Ravel, J., & Fricke, W. F. (2018) · Microbiome · doi:10.1186/s40168-017-0394-z · Cited by 52
Guo, Q., Thabane, L., Hall, G. B., McKinnon, M. C., Goeree, R. A., & Pullenayegum, E. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.08.012 · Cited by 46
A Framework for Improving the Quality of Research in the Biological Sciences
Casadevall, A., Ellis, L. M., Davies, E. W., McFall-Ngai, M., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.01256-16 · Cited by 46
The importance of being second
Editors, T. P. B. S. (2018) · PLoS Biology · doi:10.1371/journal.pbio.2005203 · Cited by 23
Good Enough Practices in Scientific Computing
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005510 · Cited by 8
Register your study as a new publication option
(2015) · AAAS Articles DO Group · doi:10.1126/science.caredit.a1500282 · Cited by 6
Cited by
Microbiome definition re-visited: old concepts and new challenges
Berg, G., Rybakova, D., Fischer, D., Cernava, T., Vergès, M.-C. C., Charles, T., Chen, X., Cocolin, L., Eversole, K., Corral, G. H., et al. (2020) · Microbiome · doi:10.1186/s40168-020-00875-0 · Cited by 2,275
RESCRIPt: Reproducible sequence taxonomy reference database management
(2021) · doi:10.1371/journal.pcbi.1009581 · Cited by 983
Systems biology and multi-omics integration: viewpoints from the metabolomics research community
Pinu, F. R., Beale, D. J., Paten, A. M., Kouremenos, K., Swarup, S., Schirra, H. J., & Wishart, D. (2019) · Metabolites · doi:10.3390/metabo9040076 · Cited by 723
Greengenes2 unifies microbial data in a single reference tree
McDonald, D., Jiang, Y., Balaban, M., Cantrell, K., Zhu, Q., Gonzalez, A., Morton, J. T., Nicolaou, G., Parks, D. H., Karst, S. M., et al. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01845-1 · Cited by 681
Cullin, N., Azevedo Antunes, C., Straussman, R., Stein-Thoeringer, C. K., & Elinav, E. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.08.006 · Cited by 578
“However, amplification biases, chimera generation, and methodological variation in amplified regions of the 16S rRNAgene, data curation, and referencedatabases, introduce challenges in analytical outputs, data interpretation, and reproducibility across studies (Schloss, 2018).”
Background
Metadata
- Authors
- Schloss, P. D.
- Venue
- mBio
- Publisher
- American Society for Microbiology
- Published
- 2018-07-05
- Volume
- 9
- Issue
- 3
- ISSN
- 2150-7511 (ISSN-L)
- 2161-2129
- OpenAlex
- W2805044645
- Cited by
- 280
- References
- 67
- FWCI
- 36.96
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.