Assessing population structure: FST and related measures

Abstract

Although FST is widely used as a measure of population structure, it has been criticized recently because of its dependency on within‐population diversity. This dependency can lead to difficulties in interpretation and in the comparison of estimates among species or among loci and has led to the development of two replacement statistics, F′ST and D. F′ST is the normal FST standardized by the maximum value it can obtain, given the observed within‐population diversity. D uses a multiplicative partitioning of diversity, based on the effective number of alleles rather than on the expected heterozygosity. In this study, we review the relationships between the three classes of statistics (FST, F′ST and D), their estimation and their properties. We illustrate the relationships between the statistics using a data set of estimates from 84 species taken from the last 4 years of Molecular Ecology. As with FST, unbiased estimators are available for the two new statistics D and F′ST. Here, we develop a new unbiased F′ST estimator based on GST, which we call G′′ST. However, F′ST can be calculated using any FST estimator for which the maximum value can be obtained. As all three statistics have their advantages and their drawbacks, we recommend continued use of FST in combination with either F′ST or D. In most cases, F′ST would be the best choice among the latter two as it is most suited for inferences of the influence of demographic processes such as genetic drift and migration on genetic population structure.