Assessment of variation in microbial community amplicon sequencing by the Microbiome Quality Control (MBQC) project consortium

May combine two claims.

Effects from data generation and bioinformatics can vary by sample and sometimes interact.

“both types of effects can differ per sample (and sometimes interact)”

May combine two claims.

At least one OTU was reported in 342 negative control samples, with counts ranging from 1 to about 130 at low rarefaction and almost 200 when all sequences were used.

“At least one OTU was reported in 342 of these samples, with the observed number ranging from 1 to ~130 at low (1,000 sequences) rarefaction and almost 200 in all sequences”

• AssumptionRests on an assumption: Negative control buffer aliquots are expected to contain no microbial DNA or OTUs, so any OTUs detected in them indicate contamination.

May combine two claims.

Beta diversities between locally and centrally extracted samples were specimen-specific: some samples showed little effect, while others differed significantly mainly because of this single handling protocol variable.

“Beta diversities between locally- versus centrally-extracted samples were specimen-specific, with some samples showing little effect and others differing significantly primarily based on this single handling protocol variable”

• AssumptionRests on an assumption: Comparing centrally extracted DNA with locally extracted DNA isolates the effect of DNA extraction from PCR and sequencing effects.

May combine two claims.

Chimeric reads made up another roughly 1-5% of mismatched reads across all laboratories, but this rate did not vary monotonically with the number of PCR thermal cycles.

“Chimeric reads represented another ~1–5% of mismatched reads across all labs, although this rate did not vary monotonically with number of PCR thermal cycles”

May combine two claims.

About 50-80% of 16S amplicon reads exactly matched one of the 20 or 22 fecal/oral reference sequences, and most laboratories varied only slightly, typically less than 10%, between samples.

“Approximately 50–80% of 16S amplicon reads matched one of the 20 or 22 fecal/oral reference sequences exactly, with most labs varying only slightly (typically <10%) between samples.”

May combine two claims.

A post-hoc mothur-based protocol gave taxonomic profiles comparable to MBQC-base bioinformatics but with a significant, modest shift: it depleted some clades dominant in protocols BL-1 to 9 and enriched otherwise less-abundant Firmicutes.

“This produced comparable taxonomic profiles to MBQC-base bioinformatics, shifted significantly (but with modest effect size) to deplete some clades that were dominant in protocols BL-1 to 9 (genus Fusobacterium, unclassified Enterobacteriaceae, most species-level assignments, etc.) and enriching the otherwise less-abundant Firmicutes”

May combine two claims.

Overall taxonomic profiles were sensitive to handling protocol, bioinformatics, and their interaction, yet OTU richness estimates were high-means around 30-35 for centrally and locally extracted samples, respectively, and near 100 in one case (HL-E handling with BL-6 bioinformatics)-despite expected values of 20 and 22 and a low rarefaction depth of 1,000 sequences.

“Overall taxonomic profiles were also sensitive to handling protocol, bioinformatics, and to the interaction of these variables, although all estimates of OTU richness were high (means ~30–35 for centrally and locally extracted samples, respectively) and neared 100 OTUs in one case (HL-E handling, BL-6 bioinformatics), despite an expectation of 20 and 22 (fecal and oral, respectively) and low raref”

May combine two claims.

Handling and bioinformatics protocol choices interacted complexly: some sample diversity estimates agreed across handling laboratories and bioinformatics protocols, while others differed significantly by handling, bioinformatics, or both.

“This resulted in complex interactions between handling and bioinformatics protocol choices: some sample diversity estimates agreed across handling labs and bioinformatics protocols, while others differed significantly across handling, bioinformatics, or both”

May combine two claims.

The most abundant contaminant OTU was an unidentified Bacteroides at 1.6 s.d. 7.0%; it appeared in 22.6% of negative control samples and was also abundant in stool samples, averaging 1.2 s.d. 3.7%.

“The most abundant contaminant OTU was an unidentified Bacteroides at 1.6 s.d. 7.0%, which was detected in 22.6% of negative control samples and was also abundant in stool samples (average 1.2 s.d. 3.7%).”

• AssumptionRests on an assumption: Negative control buffer aliquots are expected to contain no microbial DNA or OTUs, so any OTUs detected in them indicate contamination.

May combine two claims.

Artificial communities were minimally affected by sample handling protocols, and diversity measures such as Simpson's diversity showed no statistically significant variation associated with handling laboratory.

“Some samples, e.g. artificial communities, were minimally affected by different sample handling protocols, and there was no statistically significant variation in measures of diversity (e.g. Simpson’s diversity) associated with a handling laboratory.”

May combine two claims.

Most human- and chemostat-derived samples clustered together, whereas artificial communities and some human-derived samples formed outgroups.

“Most human- and chemostat-derived samples cluster together, with the biologically distinct artificial communities and a subset of human-derived samples forming outgroups.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The handling laboratories produced roughly 215 million sequence reads across 2,250 samples.

“handling laboratories generated a total of ~215M sequence reads spanning 2,250 samples”

May combine two claims.

The two laboratories using fully open-reference OTU calling had high variance in per-sample QC criteria, because in some samples many reads were captured only by de novo OTUs.

“The two labs using fully open-reference OTU calling found high variance in their per-sample QC criteria, because in some, but not all, samples many reads were captured only by de novo OTUs.”

May combine two claims.

Absolute diversity levels differed among laboratories, yet relative correlations between community profiles across laboratories for the 22 primary specimens remained high, with a minimum Spearman r of 0.9.

“Absolute levels of diversity varied among laboratories (Supplementary Fig. 2–3), although relative correlations between community profiles in the 22 primary specimens across laboratories remained high (min. Spearman r=0.9”

May combine two claims.

With arcsine-square root transformed phylum-level abundances as the readout, the greatest variability in microbiome profiling was attributed to biological differences between specimen sources and handling laboratories, matching the univariate whole-community models.

“Using arcsine-square root transformed phylum-level taxonomic abundances as a readout, the greatest variability in microbiome profiling was assigned to biological differences between specimen sources and handling laboratories, as in the univariate whole-community models above.”

May combine two claims.

Single-base errors explained many remaining reads at 0.15-0.2% per nucleotide, and two laboratories, HL-C and HL-N, two of only three using single-indexed 8nt barcodes, also produced 1-3% mismatched sequences from single-base gaps at the barcode/linker.

“Single base error accounted for many of the remainder, at a rate of 0.15–0.2% per nucleotide, while two labs (HL-C and HL-N, two of only three labs using single-indexed 8nt barcodes) also generated 1–3% mismatched sequences due to single base gaps at the barcode/linker.”

May combine two claims.

A metagenomic check of six gut and six oral artificial community samples, profiled with MetaPhlAn2 and compared with 16S amplicon abundances, supported the expected ground truth and confirmed protocol variability even for low-complexity communities.

“this experiment supported the expected ground truth and further confirmed the variability of laboratory and bioinformatics protocols when reading out even these low-complexity communities.”

• cites [22] Segata, N. et al. Metagenomic microbial community profiling using unique clade-specific marker genes. Nat. Methods 9, 811–814 (2012). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Read-level error rates did not correlate with handling laboratories' diversity averages; inflated diversity came from other handling or bioinformatics sources, though such errors reduce absolute accuracy in taxonomic profile assessment.

“read-level error rates did not correlate with handling labs’ diversity averages. Instead, inflated diversity arose from other handling (e.g. contamination) or bioinformatics sources, although such errors do decrease absolute accuracy in taxonomic profile assessment”

May combine two claims.

The three laboratories using single 8nt barcodes, HL-C, HL-N, and HL-I, had the highest proportion of these reads, though this factor was not directly associated with diversity rates or absolute accuracy.

“Three labs that used single 8nt barcodes (HL-C, HL-N, and HL-I) had the highest proportion of these reads, although this factor was not directly associated with diversity rates (see Supplementary Fig. 12) or absolute accuracy”

May combine two claims.

DNA extraction method was the individual protocol variable with the greatest effect, as also seen in univariate analyses.

“Additionally, DNA extraction method was the individual protocol variable of greatest effect, also as observed in univariate analyses.”

May combine two claims.

Among the top 100 most abundant false positive Greengenes identifiers, with average relative abundances from 0.2% to 1.5%, 34% were from genera Bacteroides, 9% were Faecalibacterium prausnitzii, 7% were genus Pseudomonas, and 6% were family Enterobacteriaceae.

“Of the top 100 most abundant false positive Greengenes identifiers (average relative abundances ranging from 0.2% to 1.5%), 34% were from genera Bacteroides; Faecalibacterium prausnitzii also comprised 9% of the top 100 IDs, genus Pseudomonas 7%, and family Enterobacteriaceae 6%”

• AssumptionRests on an assumption: Negative control buffer aliquots are expected to contain no microbial DNA or OTUs, so any OTUs detected in them indicate contamination.

Probably not a substantial claim: may not pass S3 (load-bearing).

A total of 345 negative controls were fully processed.

“In total, 345 negative controls were fully processed.”

May combine two claims.

Laboratory-specific effects interacted with bioinformatics protocols; for example, Fusobacterium was substantively detected in HL-A samples only by BL-2 and BL-6, or by BL-1, BL-4, and BL-9A in the sole HiSeq laboratory HL-L.

“Fusobacterium was only substantively detected in HL-A samples by the BL-2 and BL-6 pipelines, for example, or by BL-1, BL-4, and BL-9A in the sole HiSeq laboratory, HL-L.”

May combine two claims.

Sample handling and bioinformatics protocols affected community profile accuracy, but contamination or failure to extract some microbial DNA contributed most to variability.

“Sample handling and bioinformatics protocols had an effect on community profile accuracy (Fig. 4A–B), but the largest contributor to variability was contamination or omissions owing to microbes whose DNA was not extracted”

• AssumptionRests on an assumption: Negative control buffer aliquots are expected to contain no microbial DNA or OTUs, so any OTUs detected in them indicate contamination.

May combine two claims.

Some computational protocol differences were readily visible, while sample handling protocol differences were harder to see but still present.

“Differences induced by a subset of computational protocols are readily visible (e.g. BL-8), with those arising from sample handling protocol choices more difficult to see in this visualization but nevertheless present”

May combine two claims.

In the handling laboratory using V3-4 primers, HL-A, fecal and oral microbes were depleted across all bioinformatics protocols, while genus Lactobacillus increased.

“Fecal and oral microbes were depleted in all bioinformatics protocols in the handling lab using V3-4 primers (HL-A), in favor of genus Lactobacillus.”

May combine two claims.

Average OTU counts were 383 ±326 for human-derived samples, 325 ±244 for chemostat-derived samples, 187 ±200 for positive control artificial communities, and 302 ±283 for negative controls.

“Human-derived samples averaged 383 ±326 OTUs, with 325 ±244 in chemostat-derived samples, 187 ±200 in positive control artificial communities, and 302 ±283 in negative controls”

May combine two claims.

Quantitative alpha diversity was relatively robust to protocol differences, but bioinformatics and sample handling protocols tended to affect median observed diversity more than biological specimen type did.

“Quantitative alpha diversity was relatively robust to protocol differences (Fig. 3A), although both bioinformatics and sample handling protocols tended to affect the median observed diversity more than biological specimen type.”

May combine two claims.

Each bioinformatics laboratory caused a modest overall shift in sample microbial profiles, but only BL-8 produced a consistent, large shift.

“While each bioinformatics laboratory induced a modest overall shift in a samples’ microbial profile, only one laboratory (BL-8) resulted in a consistent, large shift in the sample composition.”

May combine two claims.

Read-level QC and OTU assignment produced an average of about 40 million reads per handling laboratory and about 35,000 ±43,700 reads per sample, totaling 15,749 filtered samples.

“The combination of read-level QC and OTU assignment yielded an average of ~40M reads per sample handling laboratory and ~35,000 ±43,700 reads/sample, for a total of 15,749 filtered samples”

May combine two claims.

Contamination sources varied: one laboratory, HL-A, had samples mainly Lactobacillus; three laboratories, HL-B, HL-F, and HL-D, had Pseudomonas, possibly from reagents; and all laboratories had sequence, bioinformatic, and/or physical contamination from gut-derived samples.

“Sources of contamination varied, including one lab with samples comprising mainly Lactobacillus (HL-A), three labs with Pseudomonas (possibly reagent contamination23 , HL-B, HL-F, and HL-D), and sequence, bioinformatic, and/or physical contamination from gut-derived samples in all laboratories.”

• cites [23] Salter, S.J. et al. Reagent and laboratory contamination can critically impact sequence-based microbiome analyses. BMC Biol. 12, 87 (2014). sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Negative control buffer aliquots are expected to contain no microbial DNA or OTUs, so any OTUs detected in them indicate contamination.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reads and QC together yielded 27,211 OTUs with Greengenes v13.5 identifiers.

“Reads and QC together produced 27,211 OTUs with Greengenes v13.5 identifiers”

May combine two claims.

Centrally extracted samples were associated with higher Bacteroidetes abundances; because all pre-extracted samples used a MO-BIO kit, this is consistent with reports that the MO-BIO based HMP protocol enriched for genera within Bacteroidetes.

“Centrally extracted (pre-extracted) samples were associated with higher abundances of Bacteroidetes; since all pre-extracted samples were extracted using a MO-BIO kit, these results are consistent with reports showing that the MO-BIO based HMP protocol generated data enriched for genera within the Bacteroidetes24.”

• cites [24] Wesolowska-Andersen, A. et al. Choice of bacterial DNA extraction method from fecal material influences community structure as evaluated by metagenomic analysis. Microbiome 2, 19 (2014). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Bacteroidetes, Firmicutes, Actinobacteria, and Proteobacteria were the four phyla present in at least half of all subjects.

“from the four phyla present in at least half of all subjects (Bacteroidetes, Firmicutes, Actinobacteria, and Proteobacteria).”

Probably not a substantial claim: may not pass S3 (load-bearing).

A single laboratory did not use paired-end reads.

“only one laboratory did not use paired end reads.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Fourteen of fifteen labs used the V4 forward primer 515F, all used reverse primer 806R, and one lab used the V3-4 forward primer 318F.

“Fourteen of the fifteen laboratories used V4 forward primer 515F (GTGCCAGCMGCCGCGGTAA) and all the laboratories used reverse primer number 806R (GGACTACHVGGGTWTCTAAT); the remaining laboratory used V3-4 forward primer 318F (ACTCCTACGGGAGGCAGCAG)6–8.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Labs stitched overlapping reads, often with filtering, using QIIME, UPARSE, FLASH, PANDAseq, or custom software.

“Overlapping reads were stitched (and often filtered) using QIIME9, UPARSE10, FLASH11, PANDAseq12, or custom software.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Three labs reported filtering OTUs from the final table for post-creation quality control, and one reported filtering samples.

“Three labs reported filtering OTUs from the resulting table for quality control after creation, and one reported filtering samples.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For OTU creation, only methods included in QIIME (6 labs) or UPARSE (4 labs) were used, though some taxonomic assignments also used RDP classification or custom software.

“Only methods included in QIIME (N=6) or UPARSE (N=4) were used for OTU creation, although taxonomic assignments in some cases also used the RDP classification 13 or custom software.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In laboratories that froze extracted DNA, storage duration ranged from 1 to 45 days.

“In the laboratories that froze the extracted DNA, it remained frozen anywhere from 1 day to 45 days.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For MiSeq runs, Illumina V2 chemistry was most common, with 2 labs using V3 and 1 using V4.

“For the MiSeq runs, Illumina chemistry V2 was used most often, with 2 laboratories using V3 chemistry and one laboratory using V4 chemistry.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sequencing read lengths varied: 4 labs used 150 bp, 1 used 175 bp, 1 used 210 bp, 6 used 250 bp, and 2 used 300 bp.

“Four laboratories used a sequencing read length of 150 base pairs, 1 used a sequencing read length of 175 base pairs, 1 used a sequencing read of 210 base pairs, 6 used a sequencing read length of 250 base pairs, and 2 used a sequencing read length of 300 base pairs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After PCR, samples were frozen in all but two laboratories, at 4 or −20 °C, for 1 to 39 days.

“After PCR amplification, samples were frozen in all laboratories except for two at 4 °C or −20 °C for anywhere from 1 day to 39 days.”

Probably not a substantial claim: may not pass S3 (load-bearing).

During sequencing, PhiX fraction ranged from 0 to 0.25.

“During sequencing, the fraction of PhiX ranged from 0 to 0.25”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most bioinformatic protocols (7) also applied a minimum or percent length threshold to retain reads, and many (5) performed explicit chimera checking.

“Most (N=7) also included a minimum (or percent) length threshold to retain reads, and many (N=5) performed explicit chimera checking as well.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The fraction of quality bases at Q=15 ranged from 0.573 to 0.961.

“the fraction of quality bases (using the Illumina default Q=15) ranged from 0.573 to 0.961”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All laboratories except one used the Illumina MiSeq for sequencing; one used the HiSeq.

“The Illumina MiSeq was used for sequencing in all laboratories except for one, which used the HiSeq;”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All bioinformatic protocols removed some raw reads by low-quality base trimming, but the exact criteria differed.

“All bioinformatic protocols removed a subset of raw reads based on low-quality base trimming, although exact criteria varied (Q threshold, from end vs. within window, etc.)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most laboratories froze extracted DNA at 4, −20, or −80 °C before amplification, while one began amplification the same day.

“After DNA was extracted, most laboratories froze the samples at 4 °C, −20 °C, or −80 °C prior to amplification, except for one which began amplification the day of DNA extraction.”

May combine two claims.

After re-blinding, nine bioinformatics laboratories analyzed the data and generated over 16,000 taxonomic profiles.

“Re-blinded data were analyzed by nine bioinformatics labs, to produce more than 16,000 taxonomic profiles”

May combine two claims.

The baseline study used blinded specimen sets from human stool, chemostats, and artificial communities, sequenced across 15 laboratories and analyzed with nine bioinformatics protocols.

“Blinded specimen sets from human stool, chemostats and artificial microbial communities were sequenced by 15 laboratories and analyzed using nine bioinformatics protocols.”

Classified as probably not a substantial claim. May combine two claims.

In the baseline MBQC, volunteer laboratories handled samples that came from a central repository and were blinded.

“This baseline MBQC was carried out by volunteer laboratories; all samples were provided by a central repository, blinded, and handled by 15 laboratories.”

May combine two claims.

Simulations found near-balanced replicates/triplicates and raw aliquots to be best powered, and this determined the final aliquot set design.

“Near-balanced replicates/triplicates (54 vs. 42 aliquots, respectively) and raw aliquots (53 raw vs. 43 pre-extracted) were the best-powered in our simulations and hence yielded the final aliquot set design.”

May combine two claims.

Absolute diversity estimates are not comparable across laboratories even with the same bioinformatics pipeline, while relative diversity levels are on average consistent among similar sample types.

“estimates of absolute diversity are not comparable among laboratories even when processed using the same bioinformatics pipeline, whereas relative diversity levels are on average consistent among similar sample types.”

Classified as probably not a substantial claim. May combine two claims.

The final 96-aliquot set contained 60% duplicates and 40% triplicates.

“The resulting combination of specimens and extracted DNA comprised 96 aliquots with 60% duplicates and 40% triplicates.”

Classified as probably not a substantial claim. May combine two claims.

Each handling laboratory received one or more blinded copies of the 96-aliquot set and processed them under a small number of fixed constraints.

“Handling laboratories each received one or more blinded copies of the 96-aliquot set and processed them within a small number of fixed constraints.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Choices of 16S rRNA gene variable region primers can improve detection or differentiation of microbes; for example, skin-abundant Propionibacterium spp. are often missed by V4 primers, and vaginal Lactobacillus spp. are better differentiated when V3 is included.

“choices of 16S rRNA gene variable region primers may enable improved detection or differentiation of microbes16; for example, Propionibacterium spp. abundant on the skin are often undetected by V4 primers29, and Lactobacillus spp. in the vagina are better differentiated by including V321.”

• cites [16] Soergel, D.A., Dey, N., Knight, R. & Brenner, S.E. Selection of primers for optimal taxonomic classification of environmental 16S rRNA gene sequences. ISME J. 6, 1440–1444 (2012). sourceNot checked yet

  Not checked in the free run.

• cites [29] Nelson, M.C., Morrison, H.G., Benjamino, J., Grim, S.L. & Graf, J. Analysis, optimization and verification of Illumina-generated 16S rRNA gene amplicon surveys. PLoS One 9, e94249 (2014). sourceNot checked yet

  Not checked in the free run.

• cites [21] Fadrosh, D.W. et al. An improved dual-indexing approach for multiplexed 16S rRNA gene sequencing on the Illumina MiSeq platform. Microbiome 2, 6 (2014). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Grossly different beta-diversity profiles generally came from a subset of strikingly dissimilar samples rather than consistent differences across all samples, suggesting that a few samples may have been contaminated, mislabeled, or otherwise altered.

“Grossly different beta-diversity profiles were generally caused by a subset of samples that were strikingly dissimilar, rather than consistent differences in all of the samples, indicating that a handful of samples may have been contaminated, mislabeled, or otherwise altered in these laboratories.”

Classified as probably not a substantial claim. May combine two claims.

Fifteen laboratories performed extraction and/or 16S amplicon sequencing, and the raw sequence data were re-blinded and sent to nine bioinformatics laboratories.

“Fifteen laboratories carried out extraction and/or 16S amplicon sequencing, and the raw sequence data were re-blinded and distributed for processing to nine bioinformatics labs”

May combine two claims.

Although the MBQC-base sample size is too small to quantify the relative sizes of these effects, sequencing error, extraction efficiency, computational stringency, bar-hopping, and contamination may all play detectable and combinatorial roles in taxonomic profiling errors.

“While the MBQC-base sample size is too small to quantify the relative sizes of these effects, sequencing error, extraction efficiency, computational stringency, bar-hopping, and contamination may all have detectable and combinatorial roles in taxonomic profiling errors.”

Classified as probably not a substantial claim. May combine two claims.

Before distribution, all aliquots received blinded identifiers, preventing handling laboratories from linking aliquots to replicates or original specimens.

“All aliquots in the set were assigned blinded identifiers prior to distribution, such that handling labs received no information linking aliquots either to replicates or to specimens of origin.”

May combine two claims.

Relative diversity levels remained consistent because each protocol carries biases that affect all samples similarly.

“This agrees with our observation that relative diversity levels remain consistent (Supplementary Fig. 2–4), because each protocol will have a set of biases that affect all samples equally.”

Classified as probably not a substantial claim.

Aliquots from 22 specimens were extracted centrally with the MO-BIO PowerSoil kit.

“aliquots of the 22 specimens were extracted at a central facility using the MO-BIO PowerSoil kit.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Handling laboratories deposited demultiplexed FASTQ files at the DACC, and these files were re-blinded before being sent to bioinformatics participants.

“Each sample handling laboratory deposited demultiplexed FASTQ files containing the resulting data at the DACC, which were then re-blinded for distribution to bioinformatics participants”

May combine two claims.

Different human host cohorts may have remarkably different Gram-negative versus Gram-positive diversity, and a protocol suited to one community type may work well in one study but not another.

“different human host cohorts may harbor remarkably different Gram-negative versus Gram-positive diversity3, 30, and a protocol appropriate for one type of community may work well in one study but not in another.”

• cites [3] Yatsunenko, T. et al. Human gut microbiome viewed across age and geography. Nature 486, 222–227 (2012). sourceNot checked yet

  Not checked in the free run.

• cites [30] De Filippo, C. et al. Impact of diet in shaping gut microbiota revealed by a comparative study in children from Europe and rural Africa. Proc. Natl. Acad. Sci. USA 107, 14691–14696 (2010). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Human- and chemostat-derived samples did not separate substantially in the high-level overview, which supports the value of chemostat samples as biologically realistic positive controls.

“human- and chemostat-derived samples do not segregate substantially in this high-level overview, highlighting the latter’s value as biologically-realistic positive controls.”

May combine two claims.

A per-barcode synthesis error rate similar to the per-read error rate, about 0.2%, can explain much of this effect and can cause a non-negligible fraction of reads to be assigned to the wrong sample through barcode hopping.

“A per-barcode synthesis error rate comparable to the per-read errors above (~0.2%) is sufficient to explain much of this effect, leading to a non-negligible fraction of reads being assigned to the wrong sample through barcode hopping (“bar-hopping”).”

Classified as probably not a substantial claim.

Some specimens were centrally extracted and then replicated or triplicated into 96-sample aliquot sets sent to handling laboratories.

“A subset of specimens were centrally extracted and replicated or triplicated into 96-sample aliquot sets that were shipped to handling laboratories”

Probably not a substantial claim: may not pass S3 (load-bearing).

All sample types, metadata, and phenotypes were kept blinded from participants until the consolidated data release at the end of the baseline experiment.

“All sample types, metadata, and phenotypes remained blinded to participants until consolidated data release at the end of the baseline experiment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Benjamini-Hochberg-Yekutieli method was used to adjust p-values for fixed and random effects across all four phyla within each model.

“P-values for fixed and random effects were calculated using a likelihood-ratio test and adjusted using the Benjamini-Hochberg-Yekutieli method across all four phyla per model.”

Classified as probably not a substantial claim.

The full linear mixed model included fixed effects for pre-extraction; for collapsed specimen type; for collapsed health status; for PCR primer; for read length; for collapsed sequencing chemistry version; for PhiX fraction; for fraction quality bases; for log read count; for collapsed OTU software; for OTU clustering; for taxonomic assignment method; and for OTU filtering.

“We first fit a full model with fixed effects for pre-extraction, collapsed specimen type, collapsed health status, PCR primer, read length, collapsed sequencing chemistry version, PhiX fraction, fraction quality bases, log read count, collapsed OTU software, OTU clustering, taxonomic assignment method, and OTU filtering”

Classified as probably not a substantial claim. May combine two claims. The quote only partly states this claim.

Participating laboratories were blinded to the sample collection methods and sample types.

“Participating labs were blinded to sample collection methods or sample types.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Separate accounts and document roots were created for each bioinformatics laboratory so that no laboratory could see data uploaded by its peers.

“Separate accounts were created for each bioinformatics laboratory with separate document roots so that each laboratory would not have visibility into the data uploaded by its peers.”

Classified as probably not a substantial claim.

Outcomes were arcsine-square root transformed abundances, applied for variance stabilization.

“Arcsine-square root transformed abundances (for variance stabilization) were used as outcomes”

Classified as probably not a substantial claim.

In-house isolates representing the bacterial genera typical of oral and gut habitats, including Gram-positive and Gram-negative bacteria with varied genomic G+C content, were used to create artificial communities.

“Artificial communities were developed from in-house isolates representative of the range of bacterial genera found within the oral or gut habitats, including both Gram positive and Gram negative bacteria with a range of genomic G+C ratios (Supplementary Table 1).”

• cites Supplementary Table 1Not checked yet

  Not checked in the free run.

May combine two claims.

Many studies, including this one, agree that DNA extraction, especially bead beating, 16S rRNA gene primer selection, and negative control contamination have large effects on variation.

“many studies, including ours, agree that DNA extraction (particularly bead beating), 16S rRNA gene primer selection, and negative control contamination have large effects on variation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Previous results have been mixed on sequencing platform effects; this study included Illumina MiSeq and HiSeq 16S data, while earlier evaluations covered those, Roche 454, Ion Torrent, Pacific Biosciences, and other technologies, including amplicon and shotgun metagenomic comparisons.

“Previous results have been mixed in their characterization of sequencing platform effects; we included Illumina MiSeq and HiSeq 16S data, but earlier evaluations capture these, Roche 454, Ion Torrent, Pacific Biosciences, and other technologies (including comparisons of amplicon and shotgun metagenomic sequencing).”

Classified as probably not a substantial claim. May combine two claims. The quote only partly states this claim.

Sample sets were designed by simulation under constraints on the proportion of pre-extracted DNA, duplicates, triplicates, fresh stool, healthy-subject specimens, required specimen representation, and inclusion of all controls.

“Sample sets were designed by simulation under the constraints that, of 96 total samples, 40–60% should be pre-extracted DNA (the remainder raw specimens), 40–60% should represent duplicates, and 40–60% triplicates.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Every strain in the artificial communities came from a human subject originally, and HMP reference genomes were added when available.

“All strains were originally isolated from human subjects, and HMP genome reference strains were included where available5.”

• cites [5] Vatanen, T. et al. Variation in microbiome LPS immunogenicity contributes to autoimmunity in humans. Cell 165, 842–853 (2016). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

After uploads and initial quality control, each laboratory's document root name was replaced with a random string to anonymize its dataset.

“After the data uploads were completed and an initial round of quality control performed, the document root names belonging to each laboratory were renamed to randomly generated strings so that each dataset was anonymized.”

May combine two claims.

Positive control sequencing data diverge in almost every published study, including this one, confirming that relative rather than absolute measures are comparable between protocols.

“Sequencing data for positive controls diverge in almost every published study28, including ours (Supplementary Dataset 1), which confirms that relative, not absolute, measures are comparable between protocols.”

• cites [28] Morgan, J.L., Darling, A.E. & Eisen, J.A. Metagenomic sequencing of an in vitro-simulated microbial community. PLoS One 5, e10209 (2010). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The simplified linear mixed model included a random effect for specimen number.

“and a random effect for specimen number.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The full linear mixed model also included random effects for specimen number, handling laboratory, and bioinformatics laboratory.

“and random effects for specimen number, handling laboratory, and bioinformatics laboratory.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A simplified linear mixed model was fit with fixed effects for handling laboratory, bioinformatics laboratory, and pre-extraction.

“We also fit a simplified model with fixed effects for handling laboratory, bioinformatics laboratory, and pre-extraction”

Classified as probably not a substantial claim.

The selected sample configuration near-equally maximized power to partition variance in multivariate modeling.

“one of the several sample configurations that near-equally maximized power to partition variance (as described below by multivariate modeling) was selected for construction and distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The central mothur reprocessing was run post-hoc and was not included in most systematic, blinded analyses.

“Since this protocol was run centrally post-hoc, it was not recorded formally nor included in most systematic, blinded analyses.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The freeze-dried specimens originated from an earlier study of people newly diagnosed with colorectal adenocarcinoma and comparison patients scheduled for elective surgery for non-cancer, non-gastrointestinal conditions in 1985-1989.

“newly diagnosed cases with adenocarcinoma of the colon or rectum were recruited prior to surgery and treatment during 1985–19891, 2. Controls were patients awaiting elective surgery for non-oncologic, non-gastrointestinal conditions at these hospitals during the same period.”

• cites [1] Human Microbiome Project Consortium. Structure, function and diversity of the healthy human microbiome. Nature 486, 207–214 (2012). sourceNot checked yet

  Not checked in the free run.

• cites [2] Qin, J. et al. A human gut microbial gene catalogue established by metagenomic sequencing. Nature 464, 59–65 (2010). sourceNot checked yet

  Not checked in the free run.