Best practices for benchmarking germline small-variant calls in human genomes
Open accessPublished version of a preprint
Abstract
Standardized benchmarking approaches are required to assess the accuracy of variants called from sequence data. Although variant-calling tools and the metrics used to assess their performance continue to improve, important challenges remain. Here, as part of the Global Alliance for Genomics and Health (GA4GH), we present a benchmarking framework for variant calling. We provide guidance on how to match variant calls with different representations, define standard performance metrics, and stratify performance by variant type and genome context. We describe limitations of high-confidence calls and regions that can be used as truth sets (for example, single-nucleotide variant concordance of two methods is 99.7% inside versus 76.5% outside high-confidence regions). Our web-based app enables comparison of variant calls against truth sets to obtain a standardized performance report. Our approach has been piloted in the PrecisionFDA variant-calling challenges to identify the best-in-class variant-calling methods within high-confidence regions. Finally, we recommend a set of best practices for using our tools and evaluating the results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The variant call format and VCFtools
Danecek, P., Auton, A., Abecasis, G. R., Albers, C. A., Banks, E., DePristo, M. A., Handsaker, R. E., Lunter, G. A., Marth, G., Sherry, S. T., et al. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr330 · Cited by 16,008 · see citing works
Molecular Findings Among Patients Referred for Clinical Whole-Exome Sequencing
Yang, Y., Muzny, D. M., Xia, F., Niu, Z., Person, R., Ding, Y., Ward, P., Braxton, A., Wang, M., Buhay, C., et al. (2014) · JAMA · doi:10.1001/jama.2014.14601 · Cited by 1,360 · see citing works
Schneider, V. A., Graves-Lindsay, T., Howe, K., Bouk, N., Chen, H.-C., Kitts, P. A., Murphy, T. D., Pruitt, K. D., Thibaud-Nissen, F., Albracht, D., et al. (2017) · Genome Research · doi:10.1101/gr.213611.116 · Cited by 1,193 · see citing works
ACMG clinical laboratory standards for next-generation sequencing
Rehm, H. L., Bale, S. J., Bayrak‐Toydemir, P., Berg, J. S., Brown, K. K., Deignan, J. L., Friez, M. J., Funke, B., Hegde, M., & Lyon, E. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.92 · Cited by 910 · see citing works
Self-citationVariation graph toolkit improves read mapping by representing genetic variation in the reference
Garrison, E., Sirén, J., Novak, A. M., Hickey, G., Eizenga, J. M., Dawson, E. T., Jones, W., Garg, S., Markello, C., Lin, M. F., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4227 · Cited by 821 · see citing works
Extensive sequencing of seven human genomes to characterize benchmark reference materials
Zook, J. M., Catoe, D., McDaniel, J., Vang, L., Spies, N., Sidow, A., Weng, Z., Liu, Y., Mason, C. E., Alexander, N., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.25 · Cited by 811 · see citing works
Self-citationIntegrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls
Zook, J. M., Chapman, B., Wang, J., Mittelman, D., Hofmann, O., Hide, W., & Salit, M. (2014) · Nature Biotechnology · doi:10.1038/nbt.2835 · Cited by 800 · see citing works
Self-citationZook, J. M., Chapman, B., Wang, J. J., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.4661 · Cited by 551 · see citing works
Self-citationStandards and Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines
Roy, S., Coldren, C. D., Karunamurthy, A. D., Kip, N. S., Klee, E. W., Lincoln, S. E., Leon, A., Pullambhatla, M., Temple‐Smolkin, R., Voelkerding, K. V., et al. (2017) · Journal of Molecular Diagnostics · doi:10.1016/j.jmoldx.2017.11.003 · Cited by 469 · see citing works
Assuring the quality of next-generation sequencing in clinical laboratory practice
Gargis, A. S., Kalman, L. V., Berry, M. W., Bick, D. P., Dimmock, D., Hambuch, T. M., Lu, F., Lyon, E., Voelkerding, K. V., Zehnbauer, B. A., et al. (2012) · Nature Biotechnology · doi:10.1038/nbt.2403 · Cited by 458 · see citing works
Self-citationAn open resource for accurately benchmarking small variant and reference calls
Zook, J. M., McDaniel, J., Olson, N. D., Wagner, J., Parikh, H. M., Heaton, H., Irvine, S. A., Trigg, L., Truty, R., McLean, C. Y., et al. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0074-6 · Cited by 404 · see citing works
Self-citationUnified representation of genetic variants
Tan, A. K. S., Abecasis, G. R., & Kang, H. M. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv112 · Cited by 388 · see citing works
College of American Pathologists' Laboratory Standards for Next-Generation Sequencing Clinical Tests
Aziz, N., Zhao, Q., Bry, L., Driscoll, D. K., Funke, B., Gibson, J. S., Grody, W. W., Hegde, M., Hoeltge, G. A., Leonard, D. G. B., et al. (2014) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2014-0250-cp · Cited by 386 · see citing works
Self-citationEberle, M. A., Fritzilas, E., Krusche, P., Källberg, M., Moore, B. L., Bekritsky, M. A., Iqbal, Z., Chuang, H., Humphray, S., Halpern, A. L., et al. (2016) · Genome Research · doi:10.1101/gr.210500.116 · Cited by 356 · see citing works
Self-citationXue, Y., Ankala, A., Wilcox, W. R., & Hegde, M. (2014) · Genetics in Medicine · doi:10.1038/gim.2014.122 · Cited by 351 · see citing works
Genome graphs and the evolution of genome inference
Paten, B., Novak, A. M., Eizenga, J. M., & Garrison, E. (2017) · Genome Research · doi:10.1101/gr.214155.116 · Cited by 351 · see citing works
participants, I.-T. D. S. M. C. C., Ewing, A. D., Houlahan, K. E., Hu, Y., Ellrott, K., Caloian, C., Yamaguchi, T. N., Bare, J. C., P’ng, C., Waggott, D. M., et al. (2015) · Nature Methods · doi:10.1038/nmeth.3407 · Cited by 332 · see citing works
Self-citationA synthetic-diploid benchmark for accurate variant-calling evaluation
Li, H., Bloom, J. M., Farjoun, Y., Fleharty, M., Gauthier, L. D., Neale, B. M., & MacArthur, D. G. (2018) · Nature Methods · doi:10.1038/s41592-018-0054-7 · Cited by 231 · see citing works
Self-citationCleary, J. G., Braithwaite, R., Gaastra, K., Hilbush, B. S., Inglis, S., Irvine, S. A., Jackson, A., Littin, R., Rathod, M., Ware, D., et al. (2015) · doi:10.1101/023754 · Cited by 230 · see citing works
Self-citationA public resource facilitating clinical use of genomes
Ball, M. P., Thakuria, J. V., Zaranek, A. W., Clegg, T., Rosenbaum, A. M., Wu, X., Angrist, M. H., Bhak, J., Bobe, J. R., Callow, M. J., et al. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1201904109 · Cited by 222 · see citing works
A standardized framework for the validation and verification of clinical molecular genetic tests
for the EuroGentest Validation Group, Mattocks, C. J., Morris, M. A., Matthijs, G., Swinnen, E., Corveleyn, A., Dequeker, E., Müller, C. R., Pratt, V., & Wallace, A. (2010) · European Journal of Human Genetics · doi:10.1038/ejhg.2010.101 · Cited by 205 · see citing works
Good laboratory practices for molecular genetic testing for heritable diseases and conditions.
Chen, B., Gagnon, M., Shahangian, S., Anderson, N., Howerton, D., & Boone, J. (2009) · PubMed · Cited by 148 · see citing works
An analytical framework for optimizing variant discovery from personal genomes
Highnam, G., Wang, J. J., Kusler, D., Zook, J. M., Vijayan, V., Leibovich, N., & Mittelman, D. (2015) · Nature Communications · doi:10.1038/ncomms7275 · Cited by 101 · see citing works
Self-citationExome-wide assessment of the functional impact and pathogenicity of multinucleotide mutationsPreprint
Kaplanis, J., Akawi, N. A., Gallone, G., McRae, J. F., Prigmore, E., Wright, C. F., FitzPatrick, D. R., Firth, H. V., Barrett, J. C., Hurles, M. E., et al. (2019) · Genome Research · doi:10.1101/gr.239756.118 · Cited by 50 · see citing works
Zook, J. M., McDaniel, J., Parikh, H. M., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/281006 · Cited by 45 · see citing works
Self-citationSM a SH: a benchmarking toolkit for human genome variant calling
Talwalkar, A., Liptrap, J., Newcomb, J., Hartl, C., Terhorst, J., Curtis, K., Bresler, M., Song, Y. S., Jordan, M. I., & Patterson, D. A. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu345 · Cited by 41 · see citing works
Genome GraphsPreprint
Novak, A. M., Hickey, G., Garrison, E. P., Blum, S., Connelly, A., Dilthey, A., Eizenga, J. M., Elmohamed, M. A. S., Guthrie, S., Kahles, A., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/101378 · Cited by 34 · see citing works
VarMatch: robust matching of small variant datasets using flexible scoring schemes
Sun, C., & Medvedev, P. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw797 · Cited by 20 · see citing works
Kaplanis, J., Samocha, K. E., Wiel, L., Zhang, Z., Arvai, K. J., Eberhardt, R. Y., Gallone, G., Lelieveld, S. H., Martin, H. C., McRae, J. F., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/258723 · Cited by 19 · see citing works
New synthetic-diploid benchmark for accurate variant calling evaluationPreprint
Li, H., Bloom, J. M., Farjoun, Y., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/223297 · Cited by 13 · see citing works
An interlaboratory study of complex variant detectionPreprint
Lincoln, S. E., Zook, J. M., Chowdhury, S., Mahamdallie, S., FELLOWES, A. P., Klee, E. W., Truty, R., Huang, C., Tomson, F. L., Cleveland, M. H., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/218529 · Cited by 11 · see citing works
Self-citationUPS-indel: a Universal Positioning System for Indels
Hasan, M. S., Wu, X., Watson, L. T., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-14400-1 · Cited by 10 · see citing works
UPS-indel: a Universal Positioning System for IndelsPreprint
Hasan, M. S., Wu, X., Watson, L. T., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/133553 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Merqury: reference-free quality, completeness, and phasing assessment for genome assemblies
Rhie, A., Walenz, B. P., Koren, S., & Phillippy, A. M. (2020) · Genome Biology · doi:10.1186/s13059-020-02134-9 · Cited by 3,643 · see citing works
Wenger, A. M., Peluso, P., Rowell, W. J., Chang, P.-C., Hall, R. J., Concepcion, G. T., Ebler, J., Fungtammasan, A., Kolesnikov, A., Olson, N. D., et al. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0217-9 · Cited by 1,841 · see citing works
Self-citationA draft human pangenome reference.
Liao, W.-W., Asri, M., Ebler, J., Doerr, D., Haukness, M., Hickey, G., Lu, S., Lucas, J. K., Monlong, J., Abel, H. J., et al. (2023) · Nature · doi:10.1038/s41586-023-05896-x · Cited by 1,129 · see citing works
“The comparison between the callsets and truth set was made with RTG’s vcfeval (Cleary et al., 2015) and Illumina’s hap.py tool (Krusche et al., 2019) on confident regions of the benchmark.”
MethodSelf-citationByrska-Bishop, M., Evani, U. S., Zhao, X., Basile, A. O., Abel, H. J., Regier, A. A., Corvelo, A., Clarke, W. E., Musunuri, R., Nagulapalli, K., et al. (2022) · Cell · doi:10.1016/j.cell.2022.08.004 · Cited by 1,094 · see citing works
Genomic data in the all of us research program
The All of Us Research Program Genomics Investigators, Manuscript Writing Group, Bick, A. G., Metcalf, G. A., Mayo, K. R., Lichtenstein, L., Rura, S., Carroll, R. J., Musick, A., Linder, J. E., et al. (2024) · Nature · doi:10.1038/s41586-023-06957-x · Cited by 829 · see citing works
A complete reference genome improves analysis of human genetic variation
Aganezov, S., Yan, S. M., Soto, D. C., Kirsche, M., Zarate, S., Avdeyev, P., Taylor, D. J., Shafin, K., Shumate, A., Xiao, C., et al. (2022) · Science · doi:10.1126/science.abl3533 · Cited by 441 · see citing works
Self-citationPangenomics enables genotyping of known structural variants in 5202 diverse genomes
Sirén, J., Monlong, J., Chang, X., Novak, A. M., Eizenga, J. M., Markello, C., Sibbesen, J. A., Hickey, G., Chang, P.-C., Carroll, A., et al. (2021) · Science · doi:10.1126/science.abg8871 · Cited by 400 · see citing works
Curated variation benchmarks for challenging medically relevant autosomal genes
Wagner, J., Olson, N. D., Harris, L., McDaniel, J., Cheng, H., Fungtammasan, A., Hwang, Y.-C., Gupta, R., Wenger, A. M., Rowell, W. J., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01158-1 · Cited by 241 · see citing works
Self-citationSemi-automated assembly of high-quality diploid human reference genomes
Jarvis, E. D., Formenti, G., Rhie, A., Guarracino, A., Yang, C., Wood, J., Tracey, A., Thibaud-Nissen, F., Vollger, M. R., Porubsky, D., et al. (2022) · Nature · doi:10.1038/s41586-022-05325-5 · Cited by 209 · see citing works
Self-citationAccuracy and efficiency of germline variant calling pipelines for human genome data
Zhao, S., Agafonov, O., Azab, A., Stokowy, T., & Hovig, E. (2020) · Scientific Reports · doi:10.1038/s41598-020-77218-4 · Cited by 118 · see citing works
Opportunities and Challenges for Molecular Understanding of Ciliopathies–The 100,000 Genomes Project
Wheway, G., Genomics England Research Consortium, & Mitchison, H. M. (2019) · Frontiers in Genetics · doi:10.3389/fgene.2019.00127 · Cited by 99 · see citing works
A diploid assembly-based benchmark for variants in the major histocompatibility complex
Chin, C.-S., Wagner, J., Zeng, Q., Garrison, E., Garg, S., Fungtammasan, A., Rautiainen, M., Aganezov, S., Kirsche, M., Zarate, S., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18564-9 · Cited by 82 · see citing works
Self-citationAccelerating genomic workflows using NVIDIA Parabricks
O’Connell, K. A., Yosufzai, Z. B., Campbell, R. A., Lobb, C. J., Engelken, H. T., Gorrell, L. M., Carlson, T. B., Catana, J. J., Mikdadi, D., Bonazzi, V. R., et al. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05292-2 · Cited by 62 · see citing works
Pangenome Graph Construction from Genome Alignment with Minigraph-Cactus
Hickey, G., Monlong, J., Ebler, J., Novak, A., Eizenga, J. M., Gao, Y., Human Pangenome Reference Consortium, Marschall, T., Li, H., & Paten, B. (2022) · doi:10.1101/2022.10.06.511217 · Cited by 345 · see citing works
Sahraeian, S. M. E., Fang, L. T., Karagiannis, K., Moos, M., Smith, S., Santana-Quintero, L., Xiao, C., Colgan, M., Hong, H., Mohiyuddin, M., et al. (2022) · Genome biology · doi:10.1186/s13059-021-02592-9 · Cited by 27 · see citing works
DeepTrio: Variant Calling in Families Using Deep Learning
Brambrink, L., Kolesnikov, A., Goel, S., Nattestad, M., Yun, T., Baid, G., Yang, H., McLean, C., Shafin, K., Chang, P.-C., et al. (2021) · doi:10.1101/2021.04.05.438434 · Cited by 32 · see citing works
FixItFelix: improving genomic analysis by fixing reference errors
Behera, S., LeFaive, J., Orchard, P., Mahmoud, M., Paulin, L. F., Farek, J., Soto, D. C., Parker, S. C. J., Smith, A. V., Dennis, M. Y., et al. (2023) · Genome Biology · doi:10.1186/s13059-023-02863-7 · Cited by 28 · see citing works
Self-citationHigh performance of a GPU-accelerated variant calling tool in genome data analysis
Zhang, Q., Liu, H., & Bu, F. (2021) · doi:10.1101/2021.12.12.472266 · Cited by 13 · see citing works
- Authors
- the Global Alliance for Genomics and Health Benchmarking Team, Krusche, P., Trigg, L., Boutros, P. C., Mason, C. E., De La Vega, F. M., Moore, B. L., Gonzalez-Porta, M., Eberle, M. A., Tezak, Z., Lababidi, S., Truty, R., Asimenos, G., Funke, B., Fleharty, M., Chapman, B. A., Salit, M., & Zook, J. M.
- Affiliations
- Illumina (United Kingdom) (GB)
- Ontario Institute for Cancer Research (CA)
- Stanford Medicine (US)
- United States Food and Drug Administration (US)
- DNAnexus (United States) (US)
- Broad Institute (US)
- Harvard University (US)
- National Institute of Standards and Technology (US)
- Corresponding author
- Zook, Justin M.
- ORCID
- Venue
- Nature Biotechnology
- Published
- 2019-03-11
- Volume
- 37
- Issue
- 5
- Pages
- 555-560
- ISSN
- 0733-222X (ISSN-L)
- 1087-0156
- 1546-1696
- PMC
- PMC6699627
- OpenAlex
- W2951331911
- Cited by
- 429
- References
- 33
- Citations by year
- 2026: 51 · 2025: 58 · 2024: 71 · 2023: 57 · 2022: 62 · 2021: 52 · 2020: 42 · 2019: 29 · 2018: 6 · 2017: 1
- FWCI
- 16.70
- Type
- journal article
- Language
- en
- Update notices
- correction (2019-03-21)
- Open access
- green
- doi.org
- Licence
- Locations
- Nature Biotechnology (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- National Institute of Standards and Technology
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Cancer Genomics and Diagnostics
- Keywords
- variant calling, benchmarking, germline mutations, small variants, truth set, GA4GH, best practices
- MeSH terms
- Algorithms, Germ Cells, Humans, Software, Genome, Human, Benchmarking, Polymorphism, Single Nucleotide, Genomics, High-Throughput Nucleotide Sequencing, Exome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
