Comparison of Genetic Distance Measures Using Human SNP Genotype Data
Abstract
Quantification of the genetic distance between populations is instrumental in many genetic research initiatives, and a large number of formulas for this purpose have been proposed. However, selection of an appropriate measure for assessing genetic distance between real-world human populations that diverged as a result of mechanisms that are not fully known can be a challenging task. We compared results from nine widely used genetic distance measures to high-density whole-genome SNP genotype data obtained on individuals from 51 world populations. Using population trees and generalized analysis of molecular variance, we found that contradictory inferences could be drawn from analyses that used different distance measures. We determined the grouping of the distance measures in terms of similarity and consistency of their values using concordance, consistency, and Procrustes analyses. Overall, the Cavalli-Sforza and Edwards distance measure differed the most from the other measures. Wright's F(ST) for diploid data, the Latter and Reynolds distances, and Nei's minimum distance measures each yielded values that were most consistent with the other eight distance measures in terms of ordering populations based on genetic distance. The Cavalli-Sforza and Edwards distance and Nei's geometric distance were least consistent. Simulation studies showed that the Cavalli-Sforza and Edwards distance is relatively more sensitive in distinguishing genetically similar populations and that the Reynolds genetic distance provides the highest sensitivity for highly divergent populations. Finally, our study suggests that using the Cavalli-Sforza and Edwards distance may provide less power for studies concerning human migration history.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The choice of a distance measure can impact the interpretation of within-species ancestral relationships based on DNA sequence.
“For within-species studies assessing the ancestral relationships between populations based on DNA sequence, it has been shown that the choice of a distance measure can impact the interpretation of the results50, 128.”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [128]
Cites
PHYLIP-Phylogeny inference package (Version 3.2)
Felsenstein, J. (1989) · Cladistics · Cited by 17,848 · see citing works
ESTIMATION OF AVERAGE HETEROZYGOSITY AND GENETIC DISTANCE FROM A SMALL NUMBER OF INDIVIDUALS
Nei, M. (1978) · Genetics · doi:10.1093/genetics/89.3.583 · Cited by 11,499 · see citing works
Tree View: An application to display phylogenetic trees on personal computers
Page, R. (1996) · Computer applications in the biosciences · doi:10.1093/bioinformatics/12.4.357 · Cited by 10,223 · see citing works
Genetic Distance between Populations
Nei, M. (1972) · The American Naturalist · doi:10.1086/282771 · Cited by 9,903 · see citing works
THE GENETICAL STRUCTURE OF POPULATIONS
Wright, S. (1949) · Annals of Eugenics · doi:10.1111/j.1469-1809.1949.tb02451.x · Cited by 6,729 · see citing works
Generalized procrustes analysis
Gower, J. C. (1975) · Psychometrika · doi:10.1007/bf02291478 · Cited by 3,267 · see citing works
Genetic Structure of Human Populations
Rosenberg, N. A., Pritchard, J. K., Weber, J. L., Cann, H. M., Kídd, K. K., Анатольевич, Ж. Л., & Feldman, M. W. (2002) · Science · doi:10.1126/science.1078311 · Cited by 3,000 · see citing works
THE NUMBER OF ALLELES THAT CAN BE MAINTAINED IN A FINITE POPULATION
Kimura, M., & Crow, J. F. (1964) · Genetics · doi:10.1093/genetics/49.4.725 · Cited by 2,690 · see citing works
Accuracy of estimated phylogenetic trees from molecular data
Nei, M., Tajima, F., & Tateno, Y. (1983) · Journal of Molecular Evolution · doi:10.1007/bf02300753 · Cited by 2,524 · see citing works
Weir, B. S. (1997) · Biometrics · doi:10.2307/2533134 · Cited by 2,263 · see citing works
Phylogenetic analysis: models and estimation procedures
Winther, R. G. (2018) · Cambridge University Press eBooks · doi:10.1017/9781316276259.010 · Cited by 2,258 · see citing works
Worldwide Human Relationships Inferred from Genome-Wide Patterns of Variation
Li, J. Z., Absher, D. M., Tang, H., Southwick, A. M., Casto, A. M., Ramachandran, S., Cann, H. M., Barsh, G. S., Feldman, M. W., Cavalli-Sforza, L. L., et al. (2008) · Science · doi:10.1126/science.1153717 · Cited by 1,947 · see citing works
ESTIMATION OF THE COANCESTRY COEFFICIENT: BASIS FOR A SHORT-TERM GENETIC DISTANCE
Reynolds, J., Weir, B. S., & Cockerham, C. C. (1983) · Genetics · doi:10.1093/genetics/105.3.767 · Cited by 1,803 · see citing works
Genetic Distances and Reconstruction of Phylogenetic Trees From Microsatellite DNA
Takezaki, N., & Nei, M. (1996) · Genetics · doi:10.1093/genetics/144.1.389 · Cited by 1,299 · see citing works
Population stratification and spurious allelic association
Cardon, L. R., & Palmer, L. J. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12520-2 · Cited by 1,189 · see citing works
Ramachandran, S., Deshpande, O., Roseman, C. C., Rosenberg, N. A., Feldman, M. W., & Cavalli-Sforza, L. L. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507611102 · Cited by 1,103 · see citing works
A Human Genome Diversity Cell Line Panel
Cann, H. M., de Toma, C., Cazes, L., Legrand, M.-F., Morel, V., Piouffre, L., Bodmer, J. G., Bodmer, W. F., Bonné, B., Cambon‐Thomsen, A., et al. (2002) · Science · doi:10.1126/science.296.5566.261b · Cited by 997 · see citing works
Interrogating a high-density SNP map for signatures of natural selection.Preprint
Akey, J., Zhang, G., Zhang, K., Jin, L., & Shriver, M. (2002) · Genome Research · doi:10.1101/gr.631202 · Cited by 932 · see citing works
The effects of human population structure on large genetic association studies
Marchini, J., Cardon, L. R., Phillips, M. S., & Donnelly, P. (2004) · Nature Genetics · doi:10.1038/ng1337 · Cited by 869 · see citing works
Assessing the impact of population stratification on genetic association studies
Freedman, M., Reich, D., Penney, K., McDonald, G., Mignault, A. A., Patterson, N., Gabriel, S., Topol, E., Smoller, J., Pato, C., et al. (2004) · Nature Genetics · doi:10.1038/ng1333 · Cited by 843 · see citing works
The Dawn of Human Matrilineal Diversity
Behar, D. M., Villems, R., Soodyall, H., Blue-Smith, J., Pereira, L., Metspalu, E., Scozzari, R., Makkan, H., Tzur, S., Comas, D., et al. (2008) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2008.04.002 · Cited by 454 · see citing works
Paetkau, D., Waits, L. P., Clarkson, P. L., Craighead, L., & Strobeck, C. (1997) · Genetics · doi:10.1093/genetics/147.4.1943 · Cited by 421 · see citing works
Cockerham, C. C. (1973) · Genetics · doi:10.1093/genetics/74.4.679 · Cited by 385 · see citing works
A Geographically Explicit Genetic Model of Worldwide Human-Settlement History
Liu, H., Prugnolle, F., Manica, A., & Balloux, F. (2006) · The American Journal of Human Genetics · doi:10.1086/505436 · Cited by 376 · see citing works
Mathematics in the Archaeological and Historical Sciences.
Sneath, P. H. A., Hodson, F. R., Kendall, D. G., & Tautu, P. (1972) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344692 · Cited by 313 · see citing works
An Icelandic example of the impact of population structure on association studies
Helgason, A., Yngvadóttir, B., Hrafnkelsson, B., Gulcher, J. R., & Stefánsson, K. (2004) · Nature Genetics · doi:10.1038/ng1492 · Cited by 285 · see citing works
Statistical methods of comparing different multivariate analyses of the same data
Gower, J. C. (1971) · Rothamsted Repository (Rothamsted Repository) · Cited by 239 · see citing works
Reif, J. C., Melchinger, A. E., & Frisch, M. (2005) · Crop Science · doi:10.2135/cropsci2005.0001 · Cited by 228 · see citing works
Kimura, M. (1954) · Genetics · doi:10.1093/genetics/39.3.280 · Cited by 182 · see citing works
Evolutionary and statistical properties of three genetic distances
Kalinowski, S. T. (2002) · Molecular Ecology · doi:10.1046/j.1365-294x.2002.01520.x · Cited by 146 · see citing works
Microsatellite variation and the differentiation of modern humans
Pérez-Lezaun, A., Calafell, F., Mateu, E., Comas, D., Ruiz-Pacheco, R., & Bertranpetit, J. (1996) · Human Genetics · doi:10.1007/s004390050299 · Cited by 132 · see citing works
A Unified Approach for Assessing Agreement for Continuous and Categorical Data
Lin, L., Hedayat, A., & Wu, W. (2007) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543400701376498 · Cited by 114 · see citing works
Laval, G., SanCristobal, M., & Chevalet, C. (2002) · Genetics Selection Evolution · doi:10.1186/1297-9686-34-4-481 · Cited by 109 · see citing works
Correlations, descent measures: drift with migration and mutation.
Cockerham, C. C., & Weir, B. S. (1987) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.84.23.8512 · Cited by 105 · see citing works
Empirical Tests of the Reliability of Phylogenetic Trees Constructed With Microsatellite DNA
Takezaki, N., & Nei, M. (2008) · Genetics · doi:10.1534/genetics.107.081505 · Cited by 102 · see citing works
Latter, B. D. H. (1972) · Genetics · doi:10.1093/genetics/70.3.475 · Cited by 91 · see citing works
Generalized Analysis of Molecular Variance
Nievergelt, C. M., Libiger, O., & Schork, N. J. (2007) · PLoS Genetics · doi:10.1371/journal.pgen.0030051 · Cited by 80 · see citing works
Self-citationTills, D. (1977) · Human Heredity · doi:10.1159/000152865 · Cited by 19 · see citing works
Krzanowski, W. J. (1971) · Biometrics · doi:10.2307/2528839 · Cited by 16 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Statistical analysis strategies for association studies involving rare variants
Bansal, V., Libiger, O., Torkamani, A., & Schork, N. J. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2867 · Cited by 455 · see citing works
Self-citationMolecular tools and analytical approaches for the characterization of farm animal genetic diversity
Lenstra, J. A., Groeneveld, L. F., Eding, H., Kantanen, J., Williams, J. L., Taberlet, P., Nicolazzi, E. L., Sölkner, J., Simianer, H., Ciani, E., et al. (2012) · Animal Genetics · doi:10.1111/j.1365-2052.2011.02309.x · Cited by 157 · see citing works
Chen, S., Huang, C., Liao, G., Sun, H., Xie, Y., Wang, J., He, M., He, M., Dai, Z., Ren, X., et al. (2023) · Gut · doi:10.1136/gutjnl-2022-328428 · Cited by 71 · see citing works
Kaplan, J. M., & Winther, R. G. (2012) · Biological Theory · doi:10.1007/s13752-012-0048-0 · Cited by 61 · see citing works
Potter, K. M., Hipkins, V. D., Mahalovich, M. F., & Means, R. E. (2015) · Tree Genetics & Genomes · doi:10.1007/s11295-015-0865-y · Cited by 60 · see citing works
Potter, K. M., Jetton, R. M., Dvorak, W. S., Hipkins, V. D., Rhea, R., & Whittier, W. A. (2011) · Conservation Genetics · doi:10.1007/s10592-011-0301-2 · Cited by 43 · see citing works
Spatial graphs highlight how multi‐generational dispersal shapes landscape genetic patterns
Boulanger, E., Dalongeville, A., Andrello, M., Mouillot, D., & Manel, S. (2020) · Ecography · doi:10.1111/ecog.05024 · Cited by 41 · see citing works
Melka, M. G., & Schenkel, F. S. (2012) · BMC Research Notes · doi:10.1186/1756-0500-5-161 · Cited by 26 · see citing works
Genetic Distance Measures: Review
DOĞAN, İ., & DOĞAN, N. (2016) · Turkiye Klinikleri Journal of Biostatistics · doi:10.5336/biostatic.2015-49517 · Cited by 26 · see citing works
Long-Range Comparison between Genes and Languages Based on Syntactic Distances
Colonna, V., Boattini, A., Guardiano, C., Dall’Ara, I., Pettener, D., Longobardi, G., & Barbujani, G. (2010) · Human Heredity · doi:10.1159/000317374 · Cited by 25 · see citing works
Radko, A., & Podbielska, A. (2021) · Genes · doi:10.3390/genes12040485 · Cited by 22 · see citing works
Winiarski, K. J., Peterman, W. E., Whiteley, A. R., & McGarigal, K. (2019) · Molecular Ecology Resources · doi:10.1111/1755-0998.13089 · Cited by 21 · see citing works
Tóth, B., Khosravi, R., Ashrafzadeh, M. R., Bagi, Z., Fehér, M., Bársony, P., Kovács, G., & Kusza, S. (2020) · Genes · doi:10.3390/genes11111268 · Cited by 18 · see citing works
Identification of novel loci affecting circulating chromogranins and related peptides
Benyamin, B., Maihofer, A. X., Schork, A. J., Hamilton, B. A., Rao, F., Schmid‐Schönbein, G. W., Zhang, K., Mahata, M., Stridsberg, M., Schork, N. J., et al. (2016) · Human Molecular Genetics · doi:10.1093/hmg/ddw380 · Cited by 16 · see citing works
Self-citationType 2 diabetes mellitus: distribution of genetic markers in Kazakh population
Sikhayeva, N. S., Talzhanov, Y. A., Iskakova, A. N., Dzharmukhanov, J., Nugmanova, R., Zholdybayeva, E. V., & Ramanculov, E. M. (2018) · Clinical Interventions in Aging · doi:10.2147/cia.s156044 · Cited by 14 · see citing works
Zhang, K., Deacon, D. C., Rao, F., Schork, A. J., Fung, M. M., Waalen, J., Schork, N. J., Nievergelt, C. M., Chi, N. C., & O’Connor, D. T. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.09.025 · Cited by 13 · see citing works
Self-citationPotter, K. M., Campbell, A. R., Josserand, S., Nelson, C. D., & Jetton, R. M. (2017) · Tree Genetics & Genomes · doi:10.1007/s11295-017-1189-x · Cited by 10 · see citing works
Zhou, Z., Shao, C., Xie, J., Xu, H., Liu, Y., Zhou, Y., Liu, Z., Zhao, Z., Tang, Q., & Sun, K. (2019) · Molecular Genetics & Genomic Medicine · doi:10.1002/mgg3.1083 · Cited by 9 · see citing works
RetractedPrabhakar, V., & Liu, K. (2023) · Archives of Clinical and Biomedical Research · doi:10.26502/acbr.50170342 · Cited by 8 · see citing works
Landscape Features Fail to Explain Spatial Genetic Structure in White‐Tailed Deer Across Ohio, USA
Bauder, J. M., Anderson, C. S., Gibbs, H. L., Tonkovich, M. J., & Walter, W. D. (2021) · Journal of Wildlife Management · doi:10.1002/jwmg.22120 · Cited by 8 · see citing works
Dissimilarity based Partial Least Squares (DPLS) for genomic prediction from SNPs
Singh, P., Engel, J., Jansen, J. J., Haan, J. D., & Buydens, L. M. C. (2016) · BMC Genomics · doi:10.1186/s12864-016-2651-0 · Cited by 4 · see citing works
Probability, Populations, Phylogenetics, and Hominin Speciation
Caldararo (2018) · Human Biology · doi:10.13110/humanbiology.90.2.04 · Cited by 7 · see citing works
Resano, M., Zanetti, D., Esteban, E. M., & Moral, P. (2016) · Human Biology · doi:10.13110/humanbiology.88.4.0275 · Cited by 4 · see citing works
Georges, A., Mijangos, J. L., Kilian, A., Patel, H. R., Aitkens, M., & Gruber, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.03.22.533737 · Cited by 4 · see citing works
The common ground in landscape effects on gene flow in two newt species in an agroecosystem
Cox, K., Schepers, R., Breusegem, A. V., & Speybroeck, J. (2023) · Conservation Genetics · doi:10.1007/s10592-023-01539-w · Cited by 3 · see citing works
Comparative Study of Various Genetic Distance Measures between Populations for the ABO Gene
Chakraborty, S. (2010) · Notulae Scientia Biologicae · doi:10.15835/nsb245447 · Cited by 3 · see citing works
Stephen, J., & Kadarmideen, H. N. (2011) · InTech eBooks · doi:10.5772/22292 · Cited by 2 · see citing works
Probability, Populations, Phylogenetics, and Hominin Speciation
Caldararo, N. (2019) · Human Biology · doi:10.1353/hub.2017.0061 · Cited by 2 · see citing works
Resano, M., Zanetti, D., Esteban, E. M., & Moral, P. (2017) · Human Biology · doi:10.1353/hub.2017.0002 · Cited by 1 · see citing works
Aborty, S. C., Bhattacharjee, S. K., Datta, G. S., & Ma, S. (2012) · Notulae Botanicae Horti Agrobotanici Cluj-Napoca
Caldararo, N. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2939996
Mestizaje genético en las poblaciones humanas de Argentina
Resano, M. (2018)
- Authors
- Libiger, O., Nievergelt, C. M., & Schork, N. J.
- Affiliations
- Scripps Research Institute (US)
- University of California San Diego (US)
- Venue
- Human Biology
- Published
- 2009-08
- Volume
- 81
- Issue
- 4
- Pages
- 389-406
- ISSN
- 0018-7143 (ISSN-L)
- 1534-6617
- OpenAlex
- W2081773979
- Cited by
- 34
- References
- 39
- Citations by year
- 2023: 4 · 2021: 3 · 2020: 2 · 2019: 3 · 2018: 3 · 2017: 3 · 2016: 4 · 2015: 1 · 2013: 1 · 2012: 4
- FWCI
- 1.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Human Biology (publishedVersion)
- PubMed (publishedVersion)
- Human Biology (submittedVersion)
- Project Muse (Johns Hopkins University) (submittedVersion)
- BioOne Complete (BioOne) (submittedVersion)
- Funders
- Univerzita Karlova v Praze
- Instituto Nacional de Medicina Genómica
- National Institutes of Health (5u01da024417-02, n01mh22005-16-0-1)
- National Institute on Aging (U19AG023122)
- National Heart, Lung, and Blood Institute (U01HL064777)
- National Institute of Mental Health (N01 MH-22005, R01MH078151, P50MH081755)
- National Institute on Drug Abuse (U01 DA024417-01)
- National Center for Research Resources (1U54RR025204-01)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Forensic and Genetic Research
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic diversity and population structure
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Keywords
- human populations, F(ST), population divergence, human migration history
- MeSH terms
- Alleles, Emigration and Immigration, Gene Frequency, Genetics, Population, Genotype, Humans, Models, Genetic, Molecular Biology, Phylogeny, Genome, Human, Polymorphism, Single Nucleotide, Population Groups
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
