The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems
Open accessAbstract
Diagnosing diseases using machine learning (ML) is rapidly being adopted in microbiome studies. However, the estimated performance associated with these models is likely overoptimistic. Moreover, there is a trend toward using black box models without a discussion of the difficulty of interpreting such models when trying to identify microbial biomarkers of disease. This work represents a step toward developing more-reproducible ML practices in applying ML to microbiome research. We implement a rigorous pipeline and emphasize the importance of selecting ML models that reflect the goal of the study. These concepts are not particular to the study of human health but can also be applied to environmental microbiology studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schloss, P. D., Westcott, S. L., Ryabin, T., Hall, J. R., Hartmann, M., Hollister, E. B., Lesniewski, R. A., Oakley, B. B., Parks, D. H., Robinson, C. J., et al. (2009) · Applied and Environmental Microbiology · doi:10.1128/aem.01541-09 · Cited by 19,549
Self-citation"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Metagenomic biomarker discovery and explanation
Segata, N., Izard, J., Waldron, L., Gevers, D., Miropolsky, L., Garrett, W. S., & Huttenhower, C. (2011) · Genome Biology · doi:10.1186/gb-2011-12-6-r60 · Cited by 16,192
VSEARCH: a versatile open source tool for metagenomics
(2016) · doi:10.7717/peerj.2584 · Cited by 10,773
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Dormann, C. F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G., Marquéz, J. R. G., Gruber, B., Lafourcade, B., Leitão, P. J., et al. (2013) · Ecography · doi:10.1111/j.1600-0587.2012.07348.x · Cited by 9,998
Changes of Intestinal Functions in Liver Cirrhosis
Fukui, H., & Wiest, R. (2016) · Inflammatory Intestinal Diseases · doi:10.1159/000444436 · Cited by 6,808
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 1×
“Previous studies have emphasized the difficulty of interpreting the feature coefficients in linear models (37) and the biases introduced by computing feature importance using built-in methods (e.”
BackgroundBias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,688
Permutation importance: a corrected feature importance measure
Altmann, A., Toloşi, L., Sander, O., & Lengauer, T. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq134 · Cited by 2,843
Alterations of the human gut microbiome in liver cirrhosis
Qin, N., Yang, F., Li, A., Prifti, E., Chen, Y., Shao, L., Guo, J., Le Chatelier, E., Yao, J., Wu, L., et al. (2014) · Nature · doi:10.1038/nature13568 · Cited by 2,177
Potential of fecal microbiota for early-stage detection of colorectal cancer
Zeller, G., Tap, J., Voigt, A. Y., Sunagawa, S., Kultima, J. R., Costea, P. I., Amiot, A., Böhm, J., Brunetti, F., Habermann, N., et al. (2014) · Molecular Systems Biology · doi:10.15252/msb.20145645 · Cited by 1,375
Wirbel, J., Pyl, P. T., Kartal, E., Zych, K., Kashani, A., Milanese, A., Fleck, J. S., Voigt, A. Y., Palleja, A., Ponnudurai, R., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0406-6 · Cited by 1,373
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Self-citationThomas, A. M., Manghi, P., Asnicar, F., Pasolli, E., Armanini, F., Zolfo, M., Beghini, F., Manara, S., Karcher, N., Pozzi, C., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0405-7 · Cited by 1,030
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06560 · Cited by 1,018
Meta‐analyses of human gut microbes associated with obesity and IBD
Walters, W. A., Xu, Z., & Knight, R. (2014) · FEBS Letters · doi:10.1016/j.febslet.2014.09.039 · Cited by 836
Machine Learning Meta-analysis of Large Metagenomic Datasets: Tools and Biological Insights
Pasolli, E., Truong, D. T., Malik, F., Waldron, L., & Segata, N. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004977 · Cited by 659
The oral microbiota in colorectal cancer is distinctive and predictive
Flemer, B., Warren, R. D., Barrett, M. P., Cisek, K., Das, A., Jeffery, I. B., Hurley, E., O‘Riordain, M., Shanahan, F., & O‘Toole, P. W. (2018) · Gut · doi:10.1136/gutjnl-2017-314814 · Cited by 646
Looking for a Signal in the Noise: Revisiting Obesity and the Microbiome
Sze, M. A., & Schloss, P. D. (2016) · mBio · doi:10.1128/mbio.01018-16 · Cited by 600 · cited 1×
“Though the microbiome field is making progress to avoid some of these pitfalls, including validating their models on independent data sets (8, 19, 20) and introducing accessible and open-source ML tools (21–24), more work is needed to improve reproducibility further and minimize overestimating for model performance.”
BackgroundSelf-citationThe human gut microbiome as a screening tool for colorectal cancer
Zackular, J. P., Rogers, M. A. M., Ruffin, M. T., & Schloss, P. D. (2014) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-14-0129 · Cited by 551
Self-citationDai, Z., Coker, O. O., Nakatsu, G., Wu, W. K. K., Zhao, L., Chen, Z., Chan, F. K. L., Kristiansen, K., Sung, J. J. Y., Wong, S. H., et al. (2018) · Microbiome · doi:10.1186/s40168-018-0451-2 · Cited by 545
Supervised classification of human microbiota
Knights, D., Costello, E. K., & Knight, R. (2011) · FEMS Microbiology Reviews · doi:10.1111/j.1574-6976.2010.00251.x · Cited by 482
Persistent microbiome alterations modulate the rate of post-dieting weight regain
Thaiss, C. A., Itav, S., Rothschild, D., Meijer, M. T., Levy, M., Moresi, C., Dohnalová, L., Braverman, S., Rozin, S., Malitsky, S., et al. (2016) · Nature · doi:10.1038/nature20796 · Cited by 471
Westcott, S. L., & Schloss, P. D. (2017) · mSphere · doi:10.1128/mspheredirect.00073-17 · Cited by 420
Self-citationBaxter, N. T., Ruffin, M. T., Rogers, M. A. M., & Schloss, P. D. (2016) · Genome Medicine · doi:10.1186/s13073-016-0290-3 · Cited by 354
Self-citationInterpretML: A Unified Framework for Machine Learning Interpretability
Nori, H., Jenkins, S., Koch, P., & Caruana, R. (2019) · arXiv · doi:10.48550/arxiv.1909.09223 · Cited by 322
Wong, S. H., Kwong, T. N. Y., Chow, T.-C., Luk, A. K. C., Dai, R. Z. W., Nakatsu, G., Lam, T. Y. T., Zhang, L., Wu, J. C. Y., Chan, F. K. L., et al. (2017) · Gut · doi:10.1136/gutjnl-2016-312766 · Cited by 310
Human gut microbiome aging clock based on taxonomic profiling and deep learning
Galkin, F., Mamoshina, P., Aliper, A., Lane, E., Moskalev, V., Gladyshev, V. N., & Zhavoronkov, A. (2020) · iScience · doi:10.1016/j.isci.2020.101199 · Cited by 292
A comprehensive evaluation of multicategory classification methods for microbiomic data
Statnikov, A., Henaff, M., Narendra, V., Konganti, K., Li, Z., Yang, L., Pei, Z., Blaser, M. J., Aliferis, C. F., & Alekseyenko, A. V. (2013) · Microbiome · doi:10.1186/2049-2618-1-11 · Cited by 203
Shifts in the Fecal Microbiota Associated with Adenomatous Polyps
Hale, V. L., Chen, J., Johnson, S., Harrington, S. C., Yab, T. C., Smyrk, T. C., Nelson, H., Boardman, L. A., Druliner, B. R., Levin, T. R., et al. (2017) · Cancer Epidemiology, Biomarkers & Prevention · doi:10.1158/1055-9965.epi-16-0337 · Cited by 199
Pretreatment gut microbiome predicts chemotherapy-related bloodstream infection
Montassier, E., Al-Ghalith, G. A., Ward, T., Corvec, S., Gastinne, T., Potel, G., Moreau, P., de la Cochetiere, M. F., Batard, E., & Knights, D. (2016) · Genome Medicine · doi:10.1186/s13073-016-0301-4 · Cited by 188
Human-Associated Microbial Signatures: Examining Their Predictive Value
Knights, D., Parfrey, L. W., Zaneveld, J., Lozupone, C., & Knight, R. D. (2011) · Cell Host & Microbe · doi:10.1016/j.chom.2011.09.003 · Cited by 169
Classification of Paediatric Inflammatory Bowel Disease using Machine Learning
Mossotto, E., Ashton, J. J., Coelho, T. A. F., Beattie, R. M., MacArthur, B. D., & Ennis, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-02606-2 · Cited by 156
Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice
Rudin, C. D., & Ustun, B. (2018) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.2018.0957 · Cited by 154
Stool DNA Testing for Screening Detection of Colorectal Neoplasia in Alaska Native People
Redwood, D. G., Asay, E. D., Blake, I. D., Sacco, P. E., Christensen, C. M., Sacco, F. D., Tiesinga, J. J., Devens, M. E., Alberts, S. R., Mahoney, D. W., et al. (2016) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2015.10.008 · Cited by 107
Ai, L., Tian, H., Chen, Z., Chen, H., Xu, J., & Fang, J. (2017) · Oncotarget · doi:10.18632/oncotarget.14488 · Cited by 87
Vangay, P., Hillmann, B., & Knights, D. (2019) · GigaScience · doi:10.1093/gigascience/giz042 · Cited by 83
Sze, M. A., & Schloss, P. D. (2018) · mBio · doi:10.1128/mbio.00630-18 · Cited by 62
Self-citationAccurate data-driven prediction does not mean high reproducibility
Li, J., Liu, L., Le, T. D., & Liu, J. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0140-2 · Cited by 62
Baxter, N. T., Koumpouras, C. C., Rogers, M. A. M., Ruffin, M. T., & Schloss, P. D. (2016) · Microbiome · doi:10.1186/s40168-016-0205-y · Cited by 59
Self-citationMassive datasets and machine learning for computational biomedicine: trends and challenges
Андреевич, К. А., Pardalos, P. M., & Karakitsiou, A. (2018) · Annals of Operations Research · doi:10.1007/s10479-018-2891-2 · Cited by 58
Gut microbiome identifies risk for colorectal polyps
Dadkhah, E., Sikaroodi, M., Korman, L. Y., Hardi, R., Baybick, J. H., Hanzel, D. K., Kuehn, G., Kuehn, T., & Gillevet, P. M. (2019) · BMJ Open Gastroenterology · doi:10.1136/bmjgast-2019-000297 · Cited by 52
Sze, M. A., Topçuoğlu, B. D., Lesniak, N. A., Ruffin, M. T., & Schloss, P. D. (2019) · mBio · doi:10.1128/mbio.01454-19 · Cited by 47
Self-citationGuiding longitudinal sampling in IBD cohorts
Vázquez‐Baeza, Y., González, A., Xu, Z. Z., Washburne, A. D., Herfarth, H. H., Sartor, R. B., & Knight, R. D. (2017) · Gut · doi:10.1136/gutjnl-2017-315352 · Cited by 43
An Improved Method for Prediction of Cancer Prognosis by Network Learning
Kim, M., Oh, I., & Ahn, J. (2018) · Genes · doi:10.3390/genes9100478 · Cited by 43
Using convolutional neural networks to explore the microbiome
Reiman, D. J., Metwally, A. A., & Dai, Y. D. (2017) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2017.8037799 · Cited by 40
Deep Learning Tools for Human Microbiome Big Data
Geman, O., Chiuchisan, I., Covașă, M., Doloc, C., Milici, M.-R., & Milici, D. L. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-62521-8_21 · Cited by 17
Quinn, T. P., & Erb, I. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/847475 · Cited by 10
Sze, M. A., & Schloss, P. D. (2018) · mBio · doi:10.1128/mbio.02076-18 · Cited by 5
Self-citation
Cited by
Artificial intelligence for natural product drug discovery
Mullowney, M. W., Duncan, K. R., Elsayed, S. S., Garg, N., van der Hooft, J. J. J., Martin, N. I., Meijer, D., Terlouw, B. R., Biermann, F., Blin, K., et al. (2023) · Nature Reviews Drug Discovery · doi:10.1038/s41573-023-00774-7 · Cited by 460
Ghannam, R. B., & Techtmann, S. M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.01.028 · Cited by 278
Machine learning for microbiologists
Asnicar, F., Thomas, A. M., Passerini, A., Waldron, L., & Segata, N. (2024) · Nature Reviews Microbiology · doi:10.1038/s41579-023-00984-1 · Cited by 256
Chen, Y., Wang, H., Lu, W., Wu, T., Yuan, W., Zhu, J., Lee, Y. K., Zhao, J., Zhang, H., & Chen, W. (2022) · Gut Microbes · doi:10.1080/19490976.2021.2025016 · Cited by 94
“Recently, machine learning had been used to solve these problems because it is able to realize the interpretation of considering the interaction between flora to increase our understanding of the existing data structure.(18) A plethora of microbiome studies has applied machine learning methods to established disease d…”
BackgroundAbavisani, M., Khoshrou, A., Foroushan, S. K., Ebadpour, N., & Sahebkar, A. (2024) · Current Research in Biotechnology · doi:10.1016/j.crbiot.2024.100211 · Cited by 75
Metadata
- Authors
- Topçuoğlu, B. D., Lesniak, N. A., Ruffin, M. T., Wiens, J., & Schloss, P. D.
- Venue
- mBio
- Publisher
- American Society for Microbiology
- Published
- 2020-06-30
- Volume
- 11
- Issue
- 3
- ISSN
- 2150-7511 (ISSN-L)
- 2161-2129
- OpenAlex
- W3033330873
- Cited by
- 266
- References
- 50
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.