CoChat
Sign Up
Cited by 10Genomics and Phylogenetic StudiesGenomics and Rare DiseasesScientific Computing and Data Management

A distributed whole genome sequencing benchmark study

Open access
Corbett, Richard D., Eveleigh, Robert, Whitney, Joe, Barai, Namrata, Bourgey, Mathieu, Chuah, Eric, Johnson, Joanne, Moore, Richard A. +15 more·2020·Frontiers in Genetics
DOI 10.3389/fgene.2020.612515PMC PMC7736078
Open this paper in CoChat — free accountOpen PDF

Abstract

Population sequencing often requires collaboration across a distributed network of sequencing centers for the timely processing of thousands of samples. In such massive efforts, it is important that participating scientists can be confident that the accuracy of the sequence data produced is not affected by which center generates the data. A study was conducted across three established sequencing centers, located in Montreal, Toronto, and Vancouver, constituting Canada’s Genomics Enterprise ( www.cgen.ca ). Whole genome sequencing was performed at each center, on three genomic DNA replicates from three well-characterized cell lines. Secondary analysis pipelines employed by each site were applied to sequence data from each of the sites, resulting in three datasets for each of four variables (cell line, replicate, sequencing center, and analysis pipeline), for a total of 81 datasets. These datasets were each assessed according to multiple quality metrics including concordance with benchmark variant truth sets to assess consistent quality across all three conditions for each variable. Three-way concordance analysis of variants across conditions for each variable was performed. Our results showed that the variant concordance between datasets differing only by sequencing center was similar to the concordance for datasets differing only by replicate, using the same analysis pipeline. We also showed that the statistically significant differences between datasets result from the analysis pipeline used, which can be unified and updated as new approaches become available. We conclude that genome sequencing projects can rely on the quality and reproducibility of aggregate data generated across a network of distributed sites.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fgene.2020.612515

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Heuristic methods are often used for the initial stage of alignment to identify regions of similarity, which are then refined using more accurate methods.

    “These methods are often used for the initial stage of alignment to identify regions of similarity, which are then refined using more accurate methods [59].”

    From Whole-Genome Alignment: Methods, Challenges, and Future Directions (2024) · cites this paper as [59]

    The Methods section lists BWA-MEM as the alignment tool used at each center, but does not describe an initial heuristic search for similar regions followed by more accurate refinement. Searches for that sequence of alignment steps found no passage in our copy that addresses it. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Auton A. Abecasis G. R. Altshuler D. M. Durbin R. M. Abecasis G. R. Bentley D. R. (2015). A global reference for human genetic variation. Nature 526 68– 74. 10.1038/nature15393 26432245 PMC4750478 · DOI · Paper page
  2. Bailey M. H. Tokheim C. Porta-Pardo E. Sengupta S. Bertrand D. Weerasinghe A. (2018). Comprehensive characterization of cancer driver genes and mutations. Cell 173 371.e18– 385.e18. 10.1016/j.cell.2018.02.060 29625053 PMC6029450 · DOI · Paper page
  3. Baskurt Z. Mastromatteo S. Gong J. Wintle R. F. Scherer S. W. Strug L. J. (2020). VikNGS: a C++ variant integration kit for next generation sequencing association analysis. Bioinforma. Oxf. Engl. 36 1283– 1285. 10.1093/bioinformatics/btz716 31580400 PMC7703770 · DOI
  4. Beck S. Berner A. M. Bignell G. Bond M. Callanan M. J. Chervova O. (2018). Personal Genome Project UK (PGP-UK): a research and citizen science hybrid project in support of personalized medicine. BMC Med. Genomics 11: 108. 10.1186/s12920-018-0423-1 30482208 PMC6257975 · DOI
  5. Chen J. Li X. Zhong H. Meng Y. Du H. (2019). Systematic comparison of germline variant calling pipelines cross multiple next-generation sequencers. Sci. Rep. 9: 9345. 10.1038/s41598-019-45835-3 31249349 PMC6597787 · DOI
  6. Cleary J. G. Braithwaite R. Gaastra K. Hilbush B. S. Inglis S. Irvine S. A. (2015). Comparing variant call files for performance benchmarking of next-generation sequencing variant calling pipelines. bioRxiv [Preprint]. 10.1101/023754 · DOI · Paper page
  7. Craig D. W. Nasser S. Corbett R. Chan S. K. Murray L. Legendre C. (2016). A somatic reference standard for cancer genome sequencing. Sci. Rep. 6: 24607. 10.1038/srep24607 27094764 PMC4837349 · DOI
  8. DePristo M. A. Banks E. Poplin R. Garimella K. V. Maguire J. R. Hartl C. (2011). A framework for variation discovery and genotyping using next-generation DNA sequencing data. Nat. Genet. 43 491– 498. 10.1038/ng.806 21478889 PMC3083463 · DOI
  9. García-Alcalde F. Okonechnikov K. Carbonell J. Cruz L. M. Götz S. Tarazona S. (2012). Qualimap: evaluating next-generation sequencing alignment data. Bioinformatics 28 2678– 2679. 10.1093/bioinformatics/bts503 22914218 · DOI
  10. Jeon S. Bhak Y. Choi Y. Jeon Y. Kim S. Jang J. (2020). Korean Genome Project: 1094 Korean personal genomes with clinical information. Sci. Adv. 6: eaaz7835. 10.1126/sciadv.aaz7835 32766443 PMC7385432 · DOI
  11. Karczewski K. J. Francioli L. C. Tiao G. Cummings B. B. Alföldi J. Wang Q. (2019). Variation across 141,456 human exomes and genomes reveals the spectrum of loss-of-function intolerance across human protein-coding genes. bioRxiv [Preprint]. 10.1101/531210 · DOI · Paper page
  12. Kim S. Scheffler K. Halpern A. L. Bekritsky M. A. Noh E. Källberg M. (2018). Strelka2: fast and accurate calling of germline and somatic variants. Nat. Methods 15 591– 594. 10.1038/s41592-018-0051-x 30013048 · DOI
  13. Knoppers B. M. Harris J. R. Budin-Ljøsne I. Dove E. S. (2014). A human rights approach to an international code of conduct for genomic and clinical data sharing. Hum. Genet. 133 895– 903. 10.1007/s00439-014-1432-6 24573176 PMC4053599 · DOI
  14. Kumaran M. Subramanian U. Devarajan B. (2019). Performance assessment of variant calling pipelines using human whole exome sequencing and simulated data. BMC Bioinformatics 20: 342. 10.1186/s12859-019-2928-9 31208315 PMC6580603 · DOI
  15. Kurtzer G. M. Sochat V. Bauer M. W. (2017). Singularity: scientific containers for mobility of compute. PLoS One 12: e0177459. 10.1371/journal.pone.0177459 28494014 PMC5426675 · DOI · Paper page
  16. Li H. (2013). Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM. ArXiv [Preprint] Available online at: http://arxiv.org/abs/1303.3997 (accessed March 9, 2020). · arXiv
  17. Li H. Handsaker B. Wysoker A. Fennell T. Ruan J. Homer N. (2009). The sequence alignment/Map format and SAMtools. Bioinforma. Oxf. Engl. 25 2078– 2079. 10.1093/bioinformatics/btp352 19505943 PMC2723002 · DOI · Paper page
  18. Lionel A. C. Costain G. Monfared N. Walker S. Reuter M. S. Hosseini S. M. (2018). Improved diagnostic yield compared with targeted gene sequencing panels suggests a role for whole-genome sequencing as a first-tier genetic test. Genet. Med. Off. J. Am. Coll. Med. Genet. 20 435– 443. 10.1038/gim.2017.119 28771251 PMC5895460 · DOI · Paper page
  19. Mascalzoni D. Dove E. S. Rubinstein Y. Dawkins H. J. S. Kole A. McCormack P. (2015). International Charter of principles for sharing bio-specimens and data. Eur. J. Hum. Genet. 23 721– 728. 10.1038/ejhg.2014.197 25248399 PMC4795058 · DOI
  20. McKenna A. Hanna M. Banks E. Sivachenko A. Cibulskis K. Kernytsky A. (2010). The genome analysis toolkit: a mapreduce framework for analyzing next-generation DNA sequencing data. Genome Res. 20 1297– 1303. 10.1101/gr.107524.110 20644199 PMC2928508 · DOI · Paper page
  21. Moore R. A. Zeng T. Docking T. R. Bosdet I. Butterfield Y. S. Munro S. (2020). Sample tracking using unique sequence controls. J. Mol. Diagn. JMD 22 141– 146. 10.1016/j.jmoldx.2019.10.011 31837431 · DOI
  22. Pleasance E. Titmuss E. Williamson L. Kwan H. Culibrk L. Zhao E. Y. (2020). Pan-cancer analysis of advanced patient tumors reveals interactions between therapy and genomic landscapes. Nat. Cancer 1 452– 468. 10.1038/s43018-020-0050-6 35121966 · DOI
  23. Rahimzadeh V. Dyke S. O. M. Knoppers B. M. (2016). An international framework for data sharing: moving forward with the global alliance for genomics and health. Biopreservation Biobanking 14 256– 259. 10.1089/bio.2016.0005 27082668 · DOI
  24. Reuter M. S. Walker S. Thiruvahindrapuram B. Whitney J. Cohn I. Sondheimer N. (2018). The personal genome project Canada: findings from whole genome sequences of the inaugural 56 participants. CMAJ Can. Med. Assoc. J. 190 E126– E136. 10.1503/cmaj.171151 29431110 PMC5798982 · DOI
  25. Robasky K. Lewis N. E. Church G. M. (2014). The role of replicates for error mitigation in next-generation sequencing. Nat. Rev. Genet. 15 56– 62. 10.1038/nrg3655 24322726 PMC4103745 · DOI
  26. Scherer S. W. Lee C. Birney E. Altshuler D. M. Eichler E. E. Carter N. P. (2007). Challenges and standards in integrating surveys of structural variation. Nat. Genet. 39 S7– S15. 10.1038/ng2093 17597783 PMC2698291 · DOI
  27. Stark R. Grzelak M. Hadfield J. (2019). RNA sequencing: the teenage years. Nat. Rev. Genet. 20 631– 656. 10.1038/s41576-019-0150-2 31341269 · DOI
  28. Stavropoulos D. J. Merico D. Jobling R. Bowdin S. Monfared N. Thiruvahindrapuram B. (2016). Whole genome sequencing expands diagnostic utility and improves clinical management in pediatric medicine. NPJ Genomic Med. 1: 15012. 10.1038/npjgenmed.2015.12 28567303 PMC5447450 · DOI
  29. Thibodeau M. L. O’Neill K. Dixon K. Reisle C. Mungall K. L. Krzywinski M. (2020). Improved structural variant interpretation for hereditary cancer susceptibility using long-read sequencing. Genet. Med. 22 1892– 1897. 10.1038/s41436-020-0880-8 32624572 PMC7605438 · DOI
  30. Tom J. A. Reeder J. Forrest W. F. Graham R. R. Hunkapiller J. Behrens T. W. (2017). Identifying and mitigating batch effects in whole genome sequencing data. BMC Bioinformatics 18: 351. 10.1186/s12859-017-1756-z 28738841 PMC5525370 · DOI
  31. Trost B. Walker S. Haider S. A. Sung W. W. L. Pereira S. Phillips C. L. (2019). Impact of DNA source on genetic variant detection from human whole-genome sequencing data. J. Med. Genet. 56 809– 817. 10.1136/jmedgenet-2019-106281 31515274 PMC6929712 · DOI
  32. Turro E. Astle W. J. Megy K. Gräf S. Greene D. Shamardina O. (2020). Whole-genome sequencing of patients with rare diseases in a national health system. Nature 583 96– 102. 10.1038/s41586-020-2434-2 32581362 PMC7610553 · DOI · Paper page
  33. Van der Auwera G. A. Carneiro M. O. Hartl C. Poplin R. Del Angel G. Levy-Moonshine A. (2013). From FastQ data to high confidence variant calls: the genome analysis toolkit best practices pipeline. Curr. Protoc. Bioinform. 43 11.10.1– 11.10.33. 10.1002/0471250953.bi1110s43 25431634 PMC4243306 · DOI · Paper page
  34. Voss K. Gentry J. Auwera G. V. D. (2017). Full-stack genomics pipelining with GATK4 + WDL + cromwell. F1000Research 6: e126144 10.7490/f1000research.1114631.1 · DOI
  35. Wang Y.-C. Olson N. D. Deikus G. Shah H. Wenger A. M. Trow J. (2019). High-coverage, long-read sequencing of Han Chinese trio reference samples. Sci. Data 6: 91. 10.1038/s41597-019-0098-2 31201313 PMC6572837 · DOI
  36. Yuen R. K. C. Merico D. Bookman M. Howe J. L. Thiruvahindrapuram B. Patel R. V. (2017). Whole genome sequencing resource identifies 18 new candidate genes for autism spectrum disorder. Nat. Neurosci. 20 602– 611. 10.1038/nn.4524 28263302 PMC5501701 · DOI · Paper page

Cited by

Sorted by citations

Authors
Corbett, R. D., Eveleigh, R., Whitney, J., Barai, N., Bourgey, M., Chuah, E., Johnson, J., Moore, R. A., Moradin, N., Mungall, K. L., Pereira, S., Reuter, M. S., Thiruvahindrapuram, B., Wintle, R. F., Ragoussis, J., Strug, L. J., Herbrick, J.-A., Aziz, N., Jones, S. J. M., Lathrop, M., Scherer, S. W., Staffa, A., & Mungall, A. J.
Affiliations
  • Provincial Health Services Authority (CA)
  • McGill Genome Centre (CA)
  • University of Toronto (CA)
  • Hospital for Sick Children (CA)
Corresponding author
  • Scherer, Stephen W.
  • Mungall, Andrew J.
ORCID
Venue
Frontiers in Genetics
Publisher
Frontiers Media SA
Published
2020-12-01
Volume
11
Pages
612515-612515
ISSN
1664-8021 (ISSN-L)
DOI
10.3389/fgene.2020.612515
PMC
PMC7736078
OpenAlex
W3109964992
Cited by
10
References
3
Citations by year
2026: 1 · 2024: 2 · 2023: 1 · 2022: 6
FWCI
0.56
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Genetics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Canada Foundation for Innovation
  • Genome Canada
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
whole genome sequencing, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review