G ST and its relatives do not measure differentiation
Abstract
G ST and its relatives are often interpreted as measures of differentiation between subpopulations, with values near zero supposedly indicating low differentiation. However, G ST necessarily approaches zero when gene diversity is high, even if subpopulations are completely differentiated, and it is not monotonic with increasing differentiation. Likewise, when diversity is equated with heterozygosity, standard similarity measures formed by taking the ratio of mean within‐subpopulation diversity to total diversity necessarily approach unity when diversity is high, even if the subpopulations are completely dissimilar (no shared alleles). None of these measures can be interpreted as measures of differentiation or similarity. The derivations of these measures contain two subtle misconceptions which cause their paradoxical behaviours. Conclusions about population differentiation, gene flow, relatedness, and conservation priority will often be wrong when based on these fixation indices or similarity measures. These are not statistical issues; the problems persist even when true population frequencies are used in the calculations. Recent advances in the mathematics of diversity identify the misconceptions, and yield mathematically consistent descriptive measures of population structure which eliminate the paradoxes produced by standard measures. These measures can be directly related to the migration and mutation rates of the finite‐island model.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Flaws with measures such as Fst and Gst may prevent them from reliably capturing diversity, differences in diversity, or population differentiation in some trivial settings because of their focus on heterozygosity.
“In addition, flaws with measures such as Fst and Gst have been pointed out that may not allow them to reliably capture diversity, differences in diversity, or population differentiation in general in some of the most trivial settings, given their focus on heterozygosity.50”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [50]
The choice of a distance measure can impact the interpretation of within-species ancestral relationships based on DNA sequence.
“For within-species studies assessing the ancestral relationships between populations based on DNA sequence, it has been shown that the choice of a distance measure can impact the interpretation of the results50, 128.”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [50]
Using different q values in the construction of Δ measures for comparing genetic diversities of two or more populations has the same effect: small q values impact differences in rare variants and large q values emphasize differences in common variants.
“The use of different q values in the construction of Δ measures for the comparison of the genetic diversities of two (or more) populations will have the same effect50, 130, 131: small q values will impact differences in rare variants and large q values emphasize differences in common variants132.”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [50]
In diversity measures, q values less than 1 produce a measure that emphasizes rare variants, and q values greater than 1 produce a measure that emphasizes common variants.
“Thus, the use of q values less than 1.0 produces a measure that emphasizes rare variants and the use of λ values greater than 1.0 produces a measure that emphasizes common variants.50, 129”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [50]
Measures of nucleotide similarity for comparing DNA sequences between pairs of individuals within a species are problematic.
“Measures of nucleotide similarity for the comparison of DNA sequences between pairs of individuals within a species are also problematic for this reason.”
From Statistical analysis strategies for association studies involving rare variants (2010) · cites this paper as [50]
Cites
Analysis of Gene Diversity in Subdivided Populations
Nei, M. (1973) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.70.12.3321 · Cited by 8,885 · see citing works
Parry, G. D. R., Johnson, M. S., Bell, R. M., Edwards, R. W., & Wathern, P. (1984) · Springer eBooks · doi:10.1007/978-1-4899-3045-3_3 · Cited by 7,592 · see citing works
THE GENETICAL STRUCTURE OF POPULATIONS
Wright, S. (1949) · Annals of Eugenics · doi:10.1111/j.1469-1809.1949.tb02451.x · Cited by 6,729 · see citing works
Diversity and Evenness: A Unifying Notation and Its Consequences
Hill, M. O. (1973) · Ecology · doi:10.2307/1934352 · Cited by 6,355 · see citing works
Jost, L. (2006) · Oikos · doi:10.1111/j.2006.0030-1299.14714.x · Cited by 4,669 · see citing works
Self-citationTHE NUMBER OF ALLELES THAT CAN BE MAINTAINED IN A FINITE POPULATION
Kimura, M., & Crow, J. F. (1964) · Genetics · doi:10.1093/genetics/49.4.725 · Cited by 2,690 · see citing works
The Apportionment of Human Diversity
Lewontin, R. C. (1972) · Evolutionary Biology · doi:10.1007/978-1-4684-9063-3_14 · Cited by 2,472 · see citing works
PARTITIONING DIVERSITY INTO INDEPENDENT ALPHA AND BETA COMPONENTS
Jost, L. (2007) · Ecology · doi:10.1890/06-1736.1 · Cited by 2,230 · see citing works
Self-citationA STANDARDIZED GENETIC DIFFERENTIATION MEASURE
Hedrick, P. W. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01814.x · Cited by 1,839 · see citing works
The estimation of population differentiation with microsatellite markers
Balloux, F., & Lugon‐Moulin, N. (2002) · Molecular Ecology · doi:10.1046/j.0962-1083.2001.01436.x · Cited by 1,206 · see citing works
Estimation of fixation indices and gene diversities
Nei, M., & Chesser, R. K. (1983) · Annals of Human Genetics · doi:10.1111/j.1469-1809.1983.tb00993.x · Cited by 937 · see citing works
Group selection for a polygenic behavioral trait: estimating the degree of population subdivision.
Crow, J. F., & Aoki, K. (1984) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.81.19.6073 · Cited by 577 · see citing works
Fundamentals of Conservation Biology
Hill, C. J., & Hunter, M. L. (1996) · Journal of Animal Ecology · doi:10.2307/5790 · Cited by 519 · see citing works
Measures of divergence between populations and the effect of forces that reduce variability
Charlesworth, B. (1998) · Molecular Biology and Evolution · doi:10.1093/oxfordjournals.molbev.a025953 · Cited by 467 · see citing works
Genetic Structure and Selection in Subdivided Populations (MPB-40)
Rousset, F. (2004) · Princeton University Press eBooks · doi:10.1515/9781400847242 · Cited by 467 · see citing works
MICROSATELLITES CAN BE MISLEADING: AN EMPIRICAL AND SIMULATION STUDY
Balloux, F., Brunner, H., Lugon‐Moulin, N., Hausser, J., & Goudet, J. (2000) · Evolution · doi:10.1111/j.0014-3820.2000.tb00573.x · Cited by 313 · see citing works
Concentration in Modern Industry
Hannah, L., & Kay, J. (1977) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-02773-6 · Cited by 294 · see citing works
O'Reilly, P. T., Canino, M. F., BAILEY, K. M., & Bentzen, P. (2004) · Molecular Ecology · doi:10.1111/j.1365-294x.2004.02214.x · Cited by 206 · see citing works
A Two‐Stage Probabilistic Approach to Multiple‐Community Similarity Indices
Chao, A., Jost, L., Chiang, S.-F., et al. (2008) · Biometrics · doi:10.1111/j.1541-0420.2008.01010.x · Cited by 197 · see citing works
Self-citationNonextensive statistical mechanics: A brief introductionPreprint
Tsallis, C., Brigatti, E., & Brasileiro, C. (2016) · Cited by 173 · see citing works
Fixation Indices in Subdivided Populations
Nagylaki, T. (1998) · Genetics · doi:10.1093/genetics/148.3.1325 · Cited by 162 · see citing works
Nonextensive statistical mechanics: A brief introduction
Tsallis, C., & Brigatti, E. (2004) · Continuum Mechanics and Thermodynamics · doi:10.1007/s00161-004-0174-4 · Cited by 158 · see citing works
Basic concepts in population, quantitative, and evolutionary genetics
Bear, J. C. (1987) · Cell · doi:10.1016/0092-8674(87)90348-5 · Cited by 95 · see citing works
FST AND GST STATISTICS IN THE FINITE ISLAND MODEL
Takahata, N., & Nei, M. (1984) · Genetics · doi:10.1093/genetics/107.3.501 · Cited by 77 · see citing works
Carreras-Carbonell, J., Macpherson, E., & Pascual, M. (2006) · Molecular Ecology · doi:10.1111/j.1365-294x.2006.03003.x · Cited by 67 · see citing works
Sanetra, M., & Crozier, R. H. (2003) · Molecular Ecology · doi:10.1046/j.1365-294x.2003.01900.x · Cited by 54 · see citing works
Evolutionary theory and processes : modern horizons : papers in honour of Eviatar Nevo
Nevo, E., & Vasser, S. P. (2004) · Kluwer Academic Publishers eBooks · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Did Warfare Among Ancestral Hunter-Gatherers Affect the Evolution of Human Social Behaviors?
Bowles, S. (2009) · Science · doi:10.1126/science.1168112 · Cited by 773 · see citing works
Estimating and interpreting <i>F</i> <sub>ST</sub> : The impact of rare variants
Bhatia, G., Patterson, N., Sankararaman, S., & Price, A. L. (2013) · Genome Research · doi:10.1101/gr.154831.113 · Cited by 612 · see citing works
Statistical analysis strategies for association studies involving rare variants
Bansal, V., Libiger, O., Torkamani, A., & Schork, N. J. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2867 · Cited by 455 · see citing works
Whither Pst? The approximation of Qst by Pst in evolutionary and conservation biology
BROMMER, J. E. (2011) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2011.02268.x · Cited by 194 · see citing works
Van OPPEN, M. J. H., BONGAERTS, P., UNDERWOOD, J. N., PEPLOW, L. M., & COOPER, T. F. (2011) · Molecular Ecology · doi:10.1111/j.1365-294x.2011.05050.x · Cited by 183 · see citing works
DiBattista, J. D., Berumen, M. L., Gaither, M. R., Rocha, L. A., Eble, J. A., Choat, J. H., Craig, M. T., Skillings, D. J., & Bowen, B. W. (2013) · Journal of Biogeography · doi:10.1111/jbi.12068 · Cited by 125 · see citing works
Genetic differentiation and the evolution of cooperation in chimpanzees and humans.
Langergraber, K., Schubert, G., Rowney, C., Wrangham, R., Zommers, Z., & Vigilant, L. (2011) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2010.2592 · Cited by 67 · see citing works
Chromosomal Evolution and Patterns of Introgression in<i>Helianthus</i>
Barb, J. G., Bowers, J. E., Renaut, S., Rey, J. I., Knapp, S. J., Rieseberg, L. H., & Burke, J. M. (2014) · Genetics · doi:10.1534/genetics.114.165548 · Cited by 55 · see citing works
Mullins, J., McDevitt, A. D., Kowalczyk, R., Ruczyńska, I., Górny, M., & Wójcik, J. M. (2014) · Acta Theriologica · doi:10.1007/s13364-014-0180-2 · Cited by 34 · see citing works
Basto, M. P., Santos-Reis, M., Simões, L., Grilo, C., Cardoso, L., Cortes, H., Bruford, M. W., & Fernandes, C. (2016) · PLOS ONE · doi:10.1371/journal.pone.0145165 · Cited by 28 · see citing works
van der Ven, R. M., Flot, J.-F., Buitrago-López, C., & Kochzius, M. (2021) · Heredity · doi:10.1038/s41437-020-00379-5 · Cited by 26 · see citing works
- Authors
- JOST, L.
- Corresponding author
- JOST, LOU
- ORCID
- JOST, LOU
- Venue
- Molecular Ecology
- Publisher
- Wiley
- Published
- 2008-09
- Volume
- 17
- Issue
- 18
- Pages
- 4015-4026
- ISSN
- 0962-1083 (ISSN-L)
- 1365-294X
- OpenAlex
- W2114709340
- Cited by
- 2,435
- References
- 27
- Citations by year
- 2026: 41 · 2025: 60 · 2024: 85 · 2023: 91 · 2022: 120 · 2021: 166 · 2020: 135 · 2019: 145 · 2018: 166 · 2017: 168 · 2016: 175 · 2015: 225 · 2014: 194 · 2013: 203 · 2012: 163
- FWCI
- 55.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Molecular Ecology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic diversity and population structure
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Forensic and Genetic Research
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Evolution and Genetic Dynamics
- Keywords
- genetic differentiation, genetic diversity, heterozygosity, similarity measures, population structure, gene flow, migration rate, mutation rate, conservation priorities, relatedness
- MeSH terms
- Genetics, Population, Heterozygote, Models, Genetic, Genetic Variation, Gene Flow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
