Multi-platform assessment of transcriptome profiling using RNA-seq in the ABRF next-generation sequencing study

Proton, PGM, and 454 detected more known junctions than Illumina HiSeq despite fewer bases sequenced.

“The Proton, PGM and 454 platforms detected more known junctions despite fewer bases sequenced compared to Illumina HiSeq.”

Banding or altered coverage distributions were likely caused by use of a different library kit version at one test site.

“The data also showed “banding” or altered coverage distributions, likely caused by the use of a different library kit version at one of the test sites (C).”

In the same Illumina library, HiSeq detected 22.6 junctions/Mb and MiSeq detected 33.9 junctions/Mb.

“22.6 junctions/Mb for HiSeq vs. 33.9 junctions/Mb for MiSeq, Supplemental Fig. 26”

No platform had more than 51% of reads classified as putative duplicates.

“Yet, all platforms showed no more than 51% of reads as putative duplicates (Supplementary Fig. 24)”

Ribo-depleted RNA samples, whether degraded or not, consistently showed more uniform gene coverage than polyA-selected libraries.

“Notably, the ribo-depleted RNA samples, whether degraded or not, consistently showed more-uniform gene coverage than did polyA-selected libraries.”

Efficiency and cost for splice junction and variant detection varied widely among all platforms.

“but highly variable efficiency and cost for splice junction and variant detection between all platforms”

High read-depth platforms showed greater DEG overlap.

“Thus, although high read–depth platforms showed greater DEG overlap”

For all platforms, reported quality values were higher than empirically derived quality values based on sequence mismatches.

“Moreover, for all platforms, the reported QVs were higher than the empirically derived QVs based on sequence mismatches”

Inter-platform correlation of normalized transcript levels for samples A and B across different platforms was high, with an average R2 of 0.83.

“The inter-platform correlation was high (R2 average of 0.83) for the same samples profiled on different platforms”

Uniformity of coverage across exons is improved with long-read technology such as PacBio despite less read depth.

“However, uniformity of coverage across exons is improved with long-read technology such as PacBio (Fig. 4a and Supplementary Table 3), despite less read depth.”

Almost all samples showed a fairly similar distribution of gene coverage.

“Almost all samples showed a fairly similar distribution of coverage for genes (Fig. 2).”

Splice junction profiling showed an early plateau for detection of known junctions.

“Splice junction profiling showed an early plateau for detection of known junctions (Fig. 3e).”

454 and PacBio had the fewest putative duplicates, at 12-20%.

“with the 454 and PacBio platforms showing the fewest duplicates (12–20%).”

• AssumptionRests on an assumption: PacBio library construction does not include amplification of the final cDNA library.

The highest and most uniform coverage of full-length transcripts came from combining 3' polyA and 5'G cap enrichment with long-read PacBio technology.

“Finally, the highest and most-uniform coverage of full-length transcripts came from preparing samples with enrichment for both the 3′ polyA tail and an antibody (Ab) for the 5′ methylguanylate cap (5′G cap), combined with long-read technology (see Online Methods for Pacific Biosciences).”

Across the deep-count sequencing platforms, expression measures showed high concordance both within and between platforms.

“The results show high intra-platform and inter-platform concordance for expression measures across the deep-count platforms”

Universal STAR alignments had the lowest mapping error rate.

“since these alignments always showed the lowest mapping error rate”

Quality values across platforms ranged from Q10 to Q60.

“Results ranged from Q10 (90% accuracy) to Q60 (99.9999% accuracy) across platforms”

HiSeq showed the lowest median inter-site CV.

“with HiSeq showing the lowest median CV.”

Platforms showed a median inter-site CV of 11-39% for quantification of detected genes using normalized expression values.

“The platforms showed a median range of 11–39% inter-site CV (Coefficient of Variation) in their quantification of detected genes using normalized gene expression values”

Intra-platform correlation was even higher, with an average R2 greater than 0.86.

“and the intra-platform correlation was even higher (R2 average greater than 0.86).”

Unannotated junctions had lower concordance across platforms than known junctions.

“However, unannotated junctions have lower concordance than known junctions across platforms (Fig. 3g).”

Across platforms, substitution mismatch rates ranged from 0.6% to 7.1%.

“but the range between all platforms spanned 0.6–7.1%.”

Most platforms showed a biased quality-value distribution in the first 1-16 bases, attributable to reverse transcriptase priming.

“First, most platforms show a biased QV distribution in the first 1–16 bases, a known effect from the reverse transcriptase (RT) priming step32.”

• cites [32] Dobin, A. et al. STAR: ultrafast universal RNA-seq aligner. Bioinformatics 29, 15–21 (2013). sourceNot checked yet

  Not checked in the free run.

For HiSeq, single-base substitutions were the most common mismatch type.

“The most common type of mismatch for HiSeq was single-base substitutions”

Except for ELAND, platform-specific aligners produced better mapping rates, gene-body coverage evenness, and PrimePCR correlations than STAR applied uniformly.

“With the exception of ELAND, each platform-specific algorithm produced better mapping rates, gene-body coverage evenness and Spearman correlations with PrimePCR quantification (Supplementary Tables 2–4) when compared to STAR applied uniformly across all platforms.”

Indel mismatch rates varied widely across platforms, from 0.017% to 4.4% of all mismatches.

“Insertion/deletion (indel) type mismatch rates were also highly variable between platforms, spanning 0.017–4.4% of all mismatches observed.”

Cross-platform splicing data showed that read types dramatically influenced each platform's measure of low abundance transcripts.

“These cross-platform splicing data showed that the types of reads dramatically influenced each platform’s measure of low abundance transcripts.”

The ABRF-NGS Study's results include more than one million putative novel splice junctions observed in at least one platform.

“The ABRF-NGS Study’s results include a wealth of putative novel splice junctions, with more than one million such junctions observed in at least one platform (Gene Expression Omnibus GSE46876)”

The resulting data sets nearly double the catalog of splice junctions for these RNA standard samples.

“The resulting data sets nearly double the catalog of splice junctions for these RNA standard samples”

ERCC correlation to TaqMan on the PGM platform was similar to that on other platforms, with mean R2 of 0.78.

“these ranges of correlation to TaqMan were similar to that observed for ERCCs sequenced on the PGM, where the mean R2 = 0.78”

The dual enrichment method for PacBio libraries provided superior 5′ to 3′ coverage of sequenced transcripts.

“The dual enrichment method for PacBio libraries provided superior 5′ to 3′ coverage of the sequenced transcripts, as illustrated by comparisons across platforms for genes consistently detected by PacBio (Supplementary Fig. 37)”

Ribo-depleted libraries had 40-47% of bases mapping to introns, compared with 7-12% for polyA RNA from the same sample.

“The ribo-depleted libraries showed 40–47% of the bases mapping to introns vs. 7–12% for polyA RNA from the same sample”

Both polyA and ribo-depletion methods produced fairly consistent RNA abundance measures (FPKM), with a median FPKM difference of only 0.055 between all genes.

“Both methods produced fairly consistent measures of RNA abundance (FPKM, Online Methods), with a median FPKM difference of only 0.055 between all genes.”

Surrogate variable analysis enabled removal of latent variables from data for ribo-depleted and polyA-enriched libraries, producing nearly indistinguishable lists of DEGs.

“surrogate variable analysis enabled removal of latent variables from the data for ribo-depleted and polyA-enriched libraries, producing nearly indistinguishable lists of DEGs”

Each platform produced unique subsets of statistically significant DEGs (FDR<0.05, FC>2), accounting for 6-11% of all called DEGs detected uniquely by a platform or preparation method.

“each platform produced unique subsets (from unique systematic effects) of statistically significant DEGs (FDR < 0.05, fold change > 2, Online Methods), ranging from 6–11% of all called DEGs detected uniquely by a platform or preparation method”

Degraded RNA analysis was highly correlated (Pearson R2 >0.83) to corresponding intact samples in PrimePCR comparison.

“a comparison to the orthogonal PrimePCR dataset showed that the degraded RNA analysis was highly correlated (Pearson R2 >0.83) to the corresponding intact samples”

Few overall differences were observed between polyA and ribo-depleted preparations in gene quantification and DEG detection.

“Yet, few overall differences were observed between the polyA and ribo-depleted library preparations in gene quantification and detection of differentially expressed genes.”

Both library types had similar accuracy as measured by Matthews correlation coefficient when compared to 802 TaqMan assays.

“Both library types had similar accuracy as measured by Matthews correlation coefficient (MCC, Fig. 5d)40, 41, which is a joint measure of the assay’s sensitivity and specificity.”

• cites [40] Cui, P. et al. A comparison between ribo-minus RNA-sequencing and polyA-selected RNA-sequencing. Genomics 96, 259–265 (2010). sourceNot checked yet

  Not checked in the free run.

• cites [41] Leek, J.T., Johnson, W.E., Parker, H.S., Jaffe, A.E. & Storey, J.D. The sva package for removing batch effects and other unwanted variation in high-throughput experiments. Bioinformatics 28, 882–883 (2012). sourceNot checked yet

  Not checked in the free run.

Comparisons of platform-specific aligners with STAR showed that mapping rates, error rates and transcript coverage are larger concerns than gene quantification for inter-platform data.

“Comparisons of platform-specific aligners with STAR showed that mapping rates, error rates and transcript coverage are larger concerns when considering inter-platform data than is gene quantification”

Degraded RNA after ribo-depletion had more reads mapping to intronic areas, similar to ribo-depleted libraries from intact RNA.

“similar to ribo-depleted libraries from intact RNA, more reads mapped to intronic areas of the genome”

Comparison of gene expression quantification across intra-platform, inter-platform, inter-protocol and inter-aligner comparisons demonstrated high correlations overall.

“comparison of gene expression quantification, including results from intra-platform, inter-platform, inter-protocol and even inter-aligner comparisons, demonstrated high correlations overall”

Lower intronic reads were similarly observed for polyA RNA on other platforms.

“lower intronic reads were similarly observed for polyA RNA on the other platforms, Supplementary Fig. 32”

Comparison of polyA and ribo-depleted libraries showed different detection of nonpolyadenylated transcripts, 3′UTRs and introns.

“A comparison of polyA and ribo-depleted libraries showed different detection of nonpolyadenylated transcripts, 3′UTRs and introns”

Putative novel splice junctions displayed greater inter-platform disparity than known splice junctions.

“These putative novel splice junctions displayed greater inter-platform disparity than the known splice junctions (Fig. 3e)”

Sequencing of degraded RNA after ribo-depletion fully covered gene bodies.

“sequencing of the degraded RNA, after ribo-depletion, fully covered the gene bodies (Fig. 2)”

Gene quantification from polyA-enriched and ribo-depleted libraries was similar.

“As polyA and ribo-depleted gene quantifications were similar”

• Also stated in: § Discussion · boosted

  “results from polyA-enriched and ribo-depleted RNA libraries, and even libraries made from severely degraded RNAs, are comparable”

The ribo-depletion library method produced a read source distribution very different from polyA preparation.

“The ribo-depletion library method produced a read source distribution very different from the polyA preparation method (Fig. 5a).”

ERCC synthetic RNA controls correlated better with TaqMan in ribo-depleted libraries than in polyA-enriched libraries, with mean R2 of 0.91 and 0.82 respectively.

“the synthetic RNAs spiked into these samples (ERCC controls) performed slightly better in the ribo-depletion protocol than the polyA-enrichment protocol (mean R2 = 0.91 and 0.82, respectively)”

• Also stated in: § Discussion · plain

  “these standards performed better in ribo-depleted libraries than in polyA libraries”

Degraded RNA had lower Spearman rank-order correlation with quantitative PCR for expression differences between samples A and B.

“the degraded RNA did have a lower Spearman rank-order correlation with quantitative PCR for the expression differences detected between samples A and B”

DEGs without SVA analysis show similar but slightly lower overlap percentage and MCC.

“The corresponding DEGs without SVA analysis show similar but slightly lower overlap percentage and MCC (Supplementary Fig. 33).”

Pairwise comparison of average normalized gene expression across replicates of the two library types for four standard samples showed high Spearman correlations (A: 0.91, B: 0.93, C: 0.92, D: 0.93).

“A pairwise comparison of the average normalized gene expression across replicates of the two library types for the four standard samples showed high Spearman correlation coefficients (sample A: 0.91, B: 0.93, C: 0.92, D: 0.93).”

Overall numbers of DEGs detected between biologically distinct samples were consistent between library preparation methods.

“The overall numbers of DEGs detected between the biologically distinct samples (A vs. B, A vs. D, etc.) were also consistent between library preparation methods”

Only two inter-site RNA-seq variation studies are publicly available: MAQC-III/SEQC and GEUVADIS.

“Only two inter-site variation studies are publicly available: the MAQC-III (a.k.a. the Sequencing Quality Control Consortium, SEQC)11 study and the GEUVARDIS Consortium12.”

• cites [11] SEQC/MAQC-III Consortium. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequencing Quality Control Consortium. Nat. Biotechnol. 10.1038/nbt.2957 (24 August 2014).Not checked yet

  Not checked in the free run.

• cites [12] 't Hoen, P.A. et al. Reproducibility of high-throughput mRNA and small RNA sequencing across laboratories. Nat. Biotechnol. 31, 1015–1022 (2013). sourceNot checked yet

  Not checked in the free run.

Very little data exist on cross-site variation of expression measures for high-throughput sequencing platforms.

“For high-throughput sequencing platforms, however, very little data exist about cross-site variation of expression measures.”

Important RNA profiling parameters such as differential expression and splice variant detection have not been consistently evaluated.

“Moreover, important RNA profiling parameters such as differential expression and splice variant detection have not been consistently evaluated.”

Reduced duplication with 454 is likely because amplification occurs after template attachment to single beads, so individual molecules have less chance to spawn multiple reads.

“while the reduced duplication with 454 is likely because the amplification step takes place after template attachment to single beads, so individual molecules in the library have less chance to spawn multiple reads.”

• AssumptionRests on an assumption: In 454 library construction, amplification occurs after template attachment to single beads.

Reverse transcriptase bias can affect observed GC content and base-frequency data.

“This RT bias can also affect the observed GC content (Supplementary Figs. 7–11) and base-frequency data11,33, 34”

• cites [11] SEQC/MAQC-III Consortium. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequencing Quality Control Consortium. Nat. Biotechnol. 10.1038/nbt.2957 (24 August 2014).Not checked yet

  Not checked in the free run.

• cites [33] Hansen, K.D., Brenner, S.E. & Dudoit, S. Biases in Illumina transcriptome sequencing caused by random hexamer priming. Nucleic Acids Res. 38, e131 (2010). sourceNot checked yet

  Not checked in the free run.

• cites [34] Aird, D. et al. Analyzing and minimizing PCR amplification bias in Illumina sequencing libraries. Genome Biol. 12, R18 (2011). sourceNot checked yet

  Not checked in the free run.

No other study has used five platforms and two standardized RNA samples replicated at multiple sites.

“no other study has been conducted using five platforms and two standardized RNA samples replicated at multiple sites”

RNA-seq is expected to produce some reads from highly expressed transcripts that begin at the same nucleotide and are designated duplicates.

“RNA-seq is expected to produce some reads from highly expressed transcripts that begin at the same nucleotide and are thus designated “duplicate.””

• cites [8] Quail, M.A. et al. A tale of three next generation sequencing platforms: comparison of Ion Torrent, Pacific Biosciences and Illumina MiSeq sequencers. BMC Genomics 13, 341 (2012). sourceNot checked yet

  Not checked in the free run.

• cites [33] Hansen, K.D., Brenner, S.E. & Dudoit, S. Biases in Illumina transcriptome sequencing caused by random hexamer priming. Nucleic Acids Res. 38, e131 (2010). sourceNot checked yet

  Not checked in the free run.

Rare-isoform quantification benefits most from greater read depths, as apparent for SRP9.

“This effect was apparent for RNA splice isoforms such as SRP9 (Fig. 4a), suggesting that rare-isoform quantification benefits the most from greater read depths (such as from the Illumina HiSeq and Life Technologies Proton).”

For platforms other than 454 and PacBio, the analysis cannot distinguish whether observed duplicates come from independent transcripts or from library amplification.

“For the other platforms, this analysis cannot distinguish whether observed duplicates are due to independent transcripts or are a consequence of library amplification”

Prior microarray platform comparisons found high inter-platform concordance for detecting differentially expressed genes.

“prior work comparing microarray platforms and methods showed high levels of inter-platform concordance for the ability to detect differentially expressed genes.”

• cites [10] Shi, L. et al. The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements. Nat. Biotechnol. 24, 1151–1161 (2006). sourceNot checked yet

  Not checked in the free run.

A substantial number of previously unannotated predicted junctions are likely genuine, as they were observed using multiple platforms.

“However, a substantial number of the previously unannotated, predicted junctions are likely genuine, as they were observed using multiple platforms”

Degraded RNA, after ribo-depletion, performed as well as intact RNA libraries across multiple metrics.

“Multiple metrics indicated that the degraded RNA performed as well as the polyA-enriched or ribo-depleted libraries from intact RNA.”

• Also stated in: § Discussion · boosted

  “gene detection and quantification with highly fragmented or degraded RNA samples (from three types of degradation) is strikingly similar to intact RNA, once ribosomal RNA is removed”

Appropriate library preparation of degraded RNA can produce accurate expression measurements.

“These data indicate that although appropriate library preparation of degraded RNA can produce accurate expression measurements”

Despite overall similarities, distinct transcriptomes are represented in libraries prepared by polyA enrichment, ribo-depletion or combined polyA and 5′G cap enrichment.

“However, despite their overall similarities, distinct transcriptomes are represented in libraries prepared by polyA enrichment, ribo-depletion or combined polyA and 5′G cap enrichment”

Different alignment algorithms have different influences on comparisons between experiments depending on the metric studied.

“the use of different alignment algorithms will have different influences on comparisons between experiments depending on the metric studied”

RNA-seq of total RNA will require greater read depths for ribo-depleted libraries (~2.5X) than for polyA libraries to achieve equal coverage of exons, owing to higher rate of intron-mapped reads.

“Owing to the higher rate of intron-mapped reads, RNA-seq of total RNA will require greater read depths for ribo-depleted libraries (~2.5X) than for polyA libraries to achieve equal coverage of exons”

Similar degraded RNA results were recently reported, suggesting low quality samples can now be considered for reliable RNA-seq expression profiling.

“Similar degraded RNA results were recently reported26, suggesting that low quality samples can now be considered for reliable RNA-seq expression profiling”

• cites [26] Pareek, C.S., Smoczynski, R. & Tretyn, A. Sequencing technologies and genome sequencing. J. Appl. Genet. 52, 413–435 (2011). sourceNot checked yet

  Not checked in the free run.

This should support studies using old, degraded or fragmented RNAs, such as FFPE tissues.

“This should support studies using old, degraded or fragmented RNAs, such as those from formalin-fixed, paraffin-embedded (FFPE) tissues in clinical archives”

The different detection of nonpolyadenylated transcripts is an intentional consequence of the enrichment protocol.

“The former is an intentional consequence of the enrichment protocol”

The instruments span different chemistries, measurement techniques, and base-calling methods, which likely contribute to system-specific noise profiles.

“These instruments span different chemistries, measurement techniques (optical vs. electrical) and base-calling methods, all of which likely play roles in the system-specific noise profiles”

Broader inter-study analyses and data mining can be successfully carried out across multiple platforms despite intrinsic differences between technologies, methods and aligners.

“This result suggests broader inter-study analyses and data mining can be successfully carried out across multiple platforms despite intrinsic differences between technologies, methods and aligners”

Bioinformatics noise can be placed alongside technical and biological noise as key factors in experimental design.

“the importance of ‘bioinformatics noise’ can be placed alongside technical and biological noise as key factors in experimental design”

Given sufficient depth, ribo-depleted libraries can include almost all differentially detected genes identified by polyA preparation, without loss of sensitivity or specificity.

“Given sufficient depth of sequencing, results from ribo-depleted libraries can include almost all of the differentially detected genes identified by the polyA preparation method, without loss of sensitivity or specificity”