CoChat
Sign Up
PreprintCited by 8Scientific Computing and Data ManagementGenomics and Phylogenetic StudiesAlgorithms and Data Compression

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Analysis-ready VCF at Biobank scale using Zarr

Open access
Czech, Eric, Millar, Timothy R., Tyler, Will, White, Tom, Elsworth, Benjamin, Guez, Jérémy, Hancox, Jonny, Jeffery, Ben, Karczewski, Konrad J., Miles, Alistair, Tallman, Sam, Unneberg, Per, Wojdyla, Rafal, Zabad, Shadi, Hammerbacher, Jeff, Kelleher, Jerome·2024·GigaScience
DOI 10.1101/2024.06.11.598241PMC PMC12127038

Preprint — published as 10.1093/gigascience/giaf049

Open this paper in CoChat — free account

Abstract

Background: Variant Call Format (VCF) is the standard file format for interchanging genetic variation data and associated quality control metrics. The usual row-wise encoding of the VCF data model (either as text or packed binary) emphasises efficient retrieval of all data for a given variant, but accessing data on a field or sample basis is inefficient. Biobank scale datasets currently available consist of hundreds of thousands of whole genomes and hundreds of terabytes of compressed VCF. Row-wise data storage is fundamentally unsuitable and a more scalable approach is needed. Results: Zarr is a format for storing multi-dimensional data that is widely used across the sciences, and is ideally suited to massively parallel processing. We present the VCF Zarr specification, an encoding of the VCF data model using Zarr, along with fundamental software infrastructure for efficient and reliable conversion at scale. We show how this format is far more efficient than standard VCF based approaches, and competitive with specialised methods for storing genotype data in terms of compression ratios and single-threaded calculation performance. We present case studies on subsets of three large human datasets (Genomics England: n =78,195; Our Future Health: n =651,050; All of Us: n =245,394) along with whole genome datasets for Norway Spruce ( n =1,063) and SARS-CoV-2 ( n =4,484,157). We demonstrate the potential for VCF Zarr to enable a new generation of high-performance and cost-effective applications via illustrative examples using cloud computing and GPUs. Conclusions: Large row-encoded VCF files are a major bottleneck for current research, and storing and processing these files incurs a substantial cost. The VCF Zarr specification, building on widely-used, open-source technologies has the potential to greatly reduce these costs, and may enable a diverse ecosystem of next-generation tools for analysing genetic variation data directly from cloud-based object stores, while maintaining compatibility with existing file-oriented workflows. Key Points: VCF is widely supported, and the underlying data model entrenched in bioinformatics pipelines. The standard row-wise encoding as text (or binary) is inherently inefficient for large-scale data processing. The Zarr format provides an efficient solution, by encoding fields in the VCF separately in chunk-compressed binary format.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.06.11.598241

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 153

  1. 10.1093/bioinformatics/btr330 · DOI
  2. 10.1038/nature15393 · DOI · Paper page
  3. 10.1136/bmj.k1687 · DOI
  4. 10.1038/s41586-018-0579-z · DOI · Paper page
  5. 10.1038/S41586-021-04103-Z · DOI
  6. 10.1038/s41586-022-04965-x · DOI · Paper page
  7. UK Biobank Whole-Genome Sequencing Consortium, Li S , Carss KJ , Halldorsson BV , Cortes A. Whole-genome sequencing of half-a-million UK Biobank participants. medRxiv 2023;p. 2023–12.
  8. Cook MB , Sanderson SC , Deanfield JE , Reddington F , Roddam A , Hunter DJ , et al. Our Future Health: a unique global resource for discovery and translational research. Nature Medicine 2025;. · DOI
  9. 10.1038/s41586-023-06957-x · DOI · Paper page
  10. 10.1186/s12711-019-0522-2 · DOI
  11. 10.1093/nar/gkad840 · DOI
  12. 10.1093/jhered/esac020 · DOI
  13. 10.1371/journal.pcbi.1009123 · DOI
  14. 10.1086/519795 · DOI · Paper page
  15. 10.1186/s13742-015-0047-8 · DOI · Paper page
  16. Band G , Marchini J. BGEN: a binary file format for imputed genotype and haplotype data. BioRxiv 2018;p. 308296. · DOI
  17. 10.1016/j.ajhg.2010.11.011 · DOI · Paper page
  18. 10.1038/S41588-021-00870-7 · DOI · Paper page
  19. 10.1038/ng.3190 · DOI · Paper page
  20. 10.1016/j.ajhg.2018.07.015 · DOI
  21. 10.1038/s41588-019-0483-y · DOI · Paper page
  22. 10.1038/s41588-023-01415-w · DOI
  23. 10.1002/mpr.1608 · DOI
  24. Panoutsopoulou K , Walter K. Quality control of common and rare variants. Genetic Epidemiology: Methods and Protocols 2018;p. 25–36. · DOI
  25. 10.1038/s41586-024-07050-7 · DOI
  26. Hemstrom W , Grummer JA , Luikart G , Christie MR . Next-generation data filtering in the genomics era. Nature Reviews Genetics 2024;p. 1–18. · DOI · Paper page
  27. 10.1016/j.ajhg.2021.08.005 · DOI
  28. 10.1016/j.ajhg.2022.11.008 · DOI
  29. Williams CM , O’Connell J , Freyman WA , 23andMe Research Team, Gignoux CR , Ramachandran S , et al. Phasing millions of samples achieves near perfect accuracy, enabling parent-of-origin classification of variants. bioRxiv 2024;p. 2024–05. · DOI
  30. 10.1371/journal.pgen.1009049 · DOI
  31. 10.1038/s41588-021-00892-1 · DOI
  32. 10.1038/s41588-023-01438-3 · DOI
  33. 10.1038/s43586-021-00056-9 · DOI · Paper page
  34. 10.1093/bioinformatics/btx299 · DOI
  35. Chen Y , Dawes R , Kim HC , Stenton SL , Walker S , Ljungdahl A , et al. De novo variants in the non-coding spliceosomal snRNA gene RNU4-are a frequent cause of syndromic neurodevelopmental disorders. medRxiv 2024;p. 2024–04.
  36. 10.1109/MCSE.2021.3059437 · DOI
  37. 2016, The FAIR Guiding Principles for scientific data management and stewardship, Scientific data, 3, 1
  38. 10.1038/nn.4404 · DOI
  39. 10.1038/s41586-020-2308-7 · DOI · Paper page
  40. 10.1093/gigascience/giab008 · DOI
  41. 10.1038/s41588-018-0184-y · DOI · Paper page
  42. 10.1038/s41588-020-0621-6 · DOI
  43. 10.1186/1471-2105-14-1 · DOI
  44. 10.1038/nmeth.3654 · DOI
  45. 10.1093/bioinformatics/btv613 · DOI
  46. 10.1093/bioinformatics/btw437 · DOI
  47. 10.1093/bioinformatics/bty023 · DOI
  48. 2020, Sparse Project VCF: efficient encoding of population genotype matrices, Bioinformatics, 36, 5537
  49. 10.1093/bioinformatics/btaa290 · DOI
  50. 10.1093/bioinformatics/btab102 · DOI
  51. 10.1093/bioinformatics/btab378 · DOI
  52. 10.1093/bioinformatics/btac413 · DOI
  53. 10.1186/s13059-023-02907-y · DOI
  54. 10.1093/gigascience/giae046 · DOI
  55. 10.1186/1471-2105-13-1 · DOI
  56. 10.1093/bioinformatics/btt460 · DOI
  57. 10.1093/bioinformatics/btu495 · DOI
  58. 10.1093/bioinformatics/btz508 · DOI
  59. 10.1093/bioinformatics/btab211 · DOI
  60. DeHaas D , Pan Z , Wei X. Genotype Representation Graphs: Enabling Efficient Analysis of Biobank-Scale Data. bioRxiv 2024;. · DOI
  61. 10.1093/bioinformatics/btu014 · DOI
  62. McVean G , Kelleher J. Linkage disequilibrium, recombination and haplotype structure. Handbook of Statistical Genomics: Two Volume Set 2019;p. 51–86. · DOI
  63. Rivas MA , Chang C. Efficient storage and regression computation for population-scale genome sequencing studies. bioRxiv 2024;. · DOI
  64. 10.1371/journal.pcbi.1003153 · DOI
  65. 10.1093/nar/gkx445 · DOI
  66. 10.1038/s41591-023-02211-z · DOI
  67. 10.1186/s12859-023-05470-2 · DOI
  68. 10.1093/bioinformatics/btx145 · DOI
  69. 10.1093/bioinformatics/bts606 · DOI
  70. 10.1093/bioinformatics/btz567 · DOI
  71. 10.1186/s12859-020-3475-0 · DOI
  72. 10.1093/bioinformatics/btr509 · DOI
  73. 10.1093/bioinformatics/btu390 · DOI
  74. Nothaft FA , Massie M , Danford T , Zhang Z , Laserson U , Yeksigian C , et al. Rethinking data-intensive science using scalable analytics systems. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data; 2015. p. 631–646. · DOI
  75. 10.1093/bioinformatics/btac010 · DOI
  76. Boufea A , Finkers R , van Kaauwen M , Kramer M , Athanasiadis IN . Managing variant calling files the big data way: Using HDFS and apache parquet. In: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies; 2017. p. 219–226. · DOI
  77. 10.1089/cmb.2019.0344 · DOI
  78. 10.1126/science.add5300 · DOI
  79. Collet Y , RFC 8878: Zstandard Compression and the ‘application/zstd’ Media Type. RFC Editor; 2021. · DOI
  80. 10.1109/MCSE.2010.51 · DOI
  81. 10.1093/gigascience/giab007 · DOI
  82. Lam SK , Pitrou A , Seibert S. Numba: a LLVM-based Python JIT compiler. In: Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC; 2015. p. 1–6. · DOI
  83. 10.1038/s41586-020-2649-2 · DOI · Paper page
  84. 10.14778/3611479.3611486 · DOI
  85. 10.1093/bioinformatics/btq671 · DOI
  86. 10.1093/nar/29.1.308 · DOI · Paper page
  87. 10.1038/s41586-022-04576-6 · DOI
  88. Shi S , Rubinacci S , Hu S , Moutsianas L , Stuckey A , Need AC , et al. A Genomics England haplotype reference panel and the imputation of the UK Biobank. medRxiv 2023;. · DOI
  89. 10.1038/s41598-023-32169-4 · DOI
  90. 10.1093/braincomms/fcad244 · DOI
  91. 10.1126/science.aay5012 · DOI · Paper page
  92. 10.1038/nature12211 · DOI
  93. 10.1093/bioinformatics/btp352 · DOI · Paper page
  94. 10.1038/s41586-024-07830-1 · DOI
  95. 10.1038/s41588-023-01622-5 · DOI
  96. 10.1038/s41588-021-00862-7 · DOI
  97. 10.1093/bioinformatics/btac401 · DOI
  98. Zhan SH , Ignatieva A , Wong Y , Eaton K , Jeffery B , Palmer DS , et al. Towards pandemic-scale ancestral recombination graphs of SARS-CoV-2. bioRxiv 2023;. · DOI
  99. Hunt M , Hinrichs AS , Anderson D , Karim L , Dearlove BL , Knaggs J , et al. Addressing pandemic-wide systematic errors in the SARS-CoV-2 phylogeny. BioRxiv 2024;.
  100. 10.1093/molbev/mst010 · DOI · Paper page
  101. 10.1093/nar/13.9.3021 · DOI · Paper page
  102. Shirley M , Ma Z , Pedersen B , Wheelan S. Efficient ‘pythonic’ access to FASTA files using pyfaidx. PeerJ PrePrints 2015;. · DOI
  103. 10.1093/BIOINFORMATICS/BTZ615 · DOI
  104. Tagami D , Bisschop G , Kelleher J. tstrait: a quantitative trait simulator for ancestral recombination graphs. Bioinformatics 2024;p. btae334. · DOI
  105. 10.1109/MM.2010.41 · DOI
  106. 2014, A survey of methods for analyzing and improving GPU energy efficiency, ACM Computing Surveys (CSUR), 47, 1
  107. 10.1109/JPROC.2008.917757 · DOI
  108. 10.1002/jcc.20829 · DOI
  109. 10.1038/s42256-022-00463-x · DOI · Paper page
  110. Okuta R , Unno Y , Nishino D , Hido S , Loomis C. CuPy: A numpy-compatible library for Nvidia GPU calculations. In: Proceedings of workshop on machine learning systems (LearningSys) in the thirty-first annual conference on neural information processing systems (NIPS), vol. 6; 2017..
  111. 10.1016/j.parco.2011.09.001 · DOI
  112. 10.1038/nature14539 · DOI · Paper page
  113. 10.1038/s41586-023-06735-9 · DOI · Paper page
  114. Paszke A , Gross S , Massa F , Lerer A , Bradbury J , Chanan G , et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 2019;32.
  115. 10.1016/j.cell.2022.08.004 · DOI · Paper page
  116. 10.1093/molbev/msac034 · DOI
  117. 2025, The scalable variant call representation: enabling genetic analysis beyond one million genomes, Bioinformatics, 41, btae746
  118. 10.1093/bioinformatics/bty492 · DOI
  119. 10.1093/bioinformatics/btab087 · DOI
  120. McKinney W. Data Structures for Statistical Computing in Python. In: Stéfan van der Walt , Jarrod Millman , editors. Proceedings of the 9th Python in Science Conference; 2010. p. 56–61. · DOI
  121. 2011, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research, 12, 2825
  122. Kluyver T , Ragan-Kelley B , Pérez F , Granger B , Bussonnier M , Frederic J , et al. Jupyter Notebooks – a publishing format for reproducible computational workflows. In: Loizides F , Schmidt B , editors. Positioning and Power in Academic Publishing: Players, Agents and Agendas IOS Press; 2016. p. 87–90.
  123. 10.1038/s41592-019-0686-2 · DOI · Paper page
  124. 10.1093/BIOINFORMATICS/BTZ540 · DOI · Paper page
  125. Rand KD , Grytten I , Pavlovic M , Kanduri C , Sandve GK . BioNumPy: Fast and easy analysis of biological data with Python. bioRxiv 2022;p. 2022–12. · DOI
  126. Hou K , Gogarten S , Kim J , Hua X , Dias JA , Sun Q , et al. Admix-kit: an integrated toolkit and pipeline for genetic analyses of admixed populations. Bioinformatics 2024;p. btae148. · DOI
  127. Hoyer S , Hamman J. xarray: N-D labeled arrays and datasets in Python. Journal of Open Research Software 2017;5(5). · DOI
  128. 2015, Proceedings of the 14th python in science conference, 130, 136
  129. 10.1038/nature24995 · DOI
  130. 10.1038/s42003-022-04352-2 · DOI
  131. Folk M , Heber G , Koziol Q , Pourmal E , Robinson D. An overview of the HDF5 technology suite and its applications. In: Proceedings of the EDBT/ICDT 2011 workshop on array databases; 2011. p. 36–47. · DOI
  132. 1990, NetCDF: an interface for scientific data access, IEEE computer graphics and applications, 10, 76, 10.1109/38.56302 · DOI
  133. Ruan X , Mueller M , Liu G , Görlitz F , Fu TM , Milkie DE , et al. Image processing tools for petabyte-scale light sheet microscopy data. Nature Methods 2024;p. 1–11. · DOI
  134. 10.1038/s41592-021-01326-w · DOI
  135. 10.1175/JTECH-D-21-0106.1 · DOI
  136. Fahnestock JR , Dow DE . Mappin: A Web Native Browse Tool for the NASA JPL ITS_LIVE Project’s Ice Velocity Dataset. In: 2023 IEEE 14th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) IEEE; 2023. p. 0097–0100. · DOI
  137. OGC forms new GeoZarr Standards Working Group to establish a Zarr encoding for geospatial data;. Accessed: 2024-04-30. https://www.ogc.org/press-release/ogc-forms-new-geozarr-standards-working-group-to-establish-a-zarr-encoding-for-geospatial-data/.
  138. 10.1007/s00418-023-02209-1 · DOI
  139. 10.1016/bs.mcb.2023.01.016 · DOI
  140. 10.1038/s41467-022-32097-3 · DOI
  141. 10.1038/s41587-023-01733-8 · DOI
  142. Marconato L , Palla G , Yamauchi KA , Virshup I , Heidari E , Treis T , et al. SpatialData: an open and universal data framework for spatial omics. Nature Methods 2024;p. 1–5. · DOI
  143. Baker EA , Huang MY , Lam A , Rahim MK , Bieniosek MF , Wang B , et al. emObject: domain specific data abstraction for spatial omics. bioRxiv 2023;p. 2023–06. · DOI
  144. 10.1038/s43588-023-00544-w · DOI
  145. 10.1016/j.ajhg.2023.03.009 · DOI
  146. 2023, DivBrowse–interactive visualization and exploratory data analysis of variant call matrices, GigaScience, 12, giad025
  147. 10.1093/bioinformatics/btx057 · DOI
  148. 10.1093/genetics/iyab229 · DOI
  149. 10.1093/gbe/evae005 · DOI
  150. 10.1371/journal.pgen.1011110 · DOI
  151. Wong Y , Ignatieva A , Koskela J , Gorjanc G , Wohns AW , Kelleher J. A general and efficient representation of ancestral recombination graphs. bioRxiv 2023;. · DOI
  152. 2018, Efficient pedigree recording for fast population genetics simulation, PLoS Computational Biology, 14, 1
  153. 10.1534/genetics.120.303253 · DOI

Metadata

Authors
Czech, E., Millar, T. R., Tyler, W., White, T., Elsworth, B., Guez, J., Hancox, J., Jeffery, B., Karczewski, K. J., Miles, A., Tallman, S., Unneberg, P., Wojdyla, R., Zabad, S., Hammerbacher, J., & Kelleher, J.
Affiliations
  • Plant & Food Research (NZ)
  • Broad Institute (US)
  • Nvidia (United Kingdom) (GB)
  • University of Oxford (GB)
  • Wellcome Sanger Institute (GB)
  • Genomics England (GB)
  • Uppsala University (SE)
  • McGill University (CA)
Corresponding author
Kelleher, Jerome
ORCID
Venue
GigaScience
Publisher
openRxiv
Published
2024-06-12
Volume
14
ISSN
2047-217X (ISSN-L)
DOI
10.1101/2024.06.11.598241
PMC
PMC12127038
OpenAlex
W4410975639
Cited by
8
References
1
Citations by year
2026: 6 · 2025: 1 · 2024: 1
FWCI
6.94
Type
journal article
Language
en
Open access
Licence
Locations
  • GigaScience (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Uppsala University Publications (Uppsala University) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
Funders
  • Bill and Melinda Gates Foundation (INV-001927)
  • Wellcome Trust
  • Robertson Foundation
  • Knut och Alice Wallenbergs Stiftelse (KAW 2020.0239, KAW 2017.0003)
  • Vetenskapsrådet (2022-06725)
  • Science for Life Laboratory (KAW 2020.0239)
  • National Institutes of Health (1r56hg011395-01a1)
  • Engineering and Physical Sciences Research Council (EP/X024881/1)
  • National Human Genome Research Institute (R56 HG011395, R01HG012473)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Physical Sciences › Mathematics › Theoretical Computer Science › Algorithms and Data Compression
Keywords
variant call format, Zarr, multidimensional data storage, whole genome sequencing, data compression, cloud object storage, GPU computing, Genomics England, All of Us, Norway spruce, SARS-CoV-2

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review