Challenges in Species Tree Estimation Under the Multispecies Coalescent Model
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The multispecies coalescent (MSC) model has emerged as a powerful framework for inferring species phylogenies while accounting for ancestral polymorphism and gene tree-species tree conflict. A number of methods have been developed in the past few years to estimate the species tree under the MSC. The full likelihood methods (including maximum likelihood and Bayesian inference) average over the unknown gene trees and accommodate their uncertainties properly but involve intensive computation. The approximate or summary coalescent methods are computationally fast and are applicable to genomic datasets with thousands of loci, but do not make an efficient use of information in the multilocus data. Most of them take the two-step approach of reconstructing the gene trees for multiple loci by phylogenetic methods and then treating the estimated gene trees as observed data, without accounting for their uncertainties appropriately. In this article we review the statistical nature of the species tree estimation problem under the MSC, and explore the conceptual issues and challenges of species tree estimation by focusing mainly on simple cases of three or four closely related species. We use mathematical analysis and computer simulation to demonstrate that large differences in statistical performance may exist between the two classes of methods. We illustrate that several counterintuitive behaviors may occur with the summary methods but they are due to inefficient use of information in the data by summary methods and vanish when the data are analyzed using full-likelihood methods. These include (i) unidentifiability of parameters in the model, (ii) inconsistency in the so-called anomaly zone, (iii) singularity on the likelihood surface, and (iv) deterioration of performance upon addition of more data. We discuss the challenges and strategies of species tree inference for distantly related species when the molecular clock is violated, and highlight the need for improving the computational efficiency and model realism of the likelihood methods as well as the statistical efficiency of the summary methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evolutionary trees from DNA sequences: A maximum likelihood approach.
Felsenstein, J. (1981) · Journal of Molecular Evolution · doi:10.1007/bf01734359 · Cited by 14,571 · cited 1×
“…log X Gi Z f ðGi; tijS;QÞ fðXijGi; tiÞdti ; (2) where f(Gi, ti | S,Q) is the MSC density for the gene tree and coalescent times at locus i (Rannala and Yang 2003) and f(Xi | Gi, ti) is the probability of the sequence data given the gene tree, known as the phylogenetic likelihood (Felsenstein 1981).”
BackgroundEvolution of Protein Molecules
JUKES, T. H., & CANTOR, C. R. (1969) · Mammalian Protein Metabolism · doi:10.1016/b978-1-4832-3211-9.50009-7 · Cited by 11,032
Relaxed Phylogenetics and Dating with Confidence
Drummond, A. J., Ho, S. Y. W., Phillips, M. J., & Rambaut, A. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040088 · Cited by 6,463 · cited 1×
“Relaxed-clock models developed for dating species divergences (Thorne et al. 1998; Drummond et al. 2006; Rannala and Yang 2007) allow the substitution rate to drift over time or among branches, but they do not appear appropriate for analysis under the MSC.”
BackgroundBayesian inference of species trees from multilocus data
Heled, J., & Drummond, A. J. (2010) · Molecular Biology and Evolution · doi:10.1093/molbev/msp274 · Cited by 2,453 · cited 3×
“Full likelihood methods involve averaging over the gene trees and are computationally intensive.”
MethodEstimating the rate of evolution of the rate of molecular evolution
Thorne, J. L., Kishino, H., & Painter, I. S. (1998) · Molecular Biology and Evolution · doi:10.1093/oxfordjournals.molbev.a025892 · Cited by 1,317
A maximum pseudo-likelihood approach for estimating species trees under the coalescent model
Liu, L., Yu, L., & Edwards, S. V. (2010) · BMC Evolutionary Biology · doi:10.1186/1471-2148-10-302 · Cited by 607 · cited 15×
“This is maximum tree (MT) method of Liu et al. (2010b), implemented in the STEM program (Kubatko et al. 2009), or the GLASS algorithm (for Global LAteSt Split) (Mossel and Roch 2010).”
BackgroundMethodResultInfluentialInferring speciation times under an episodic molecular clock
Rannala, B., & Yang, Z. (2007) · Systematic Biology · doi:10.1080/10635150701420643 · Cited by 475 · cited 3×
“Rannala, B., and Z. Yang, 2007 Inferring speciation times under an episodic molecular clock.”
BackgroundMethodSelf-citationPhylogenomics reveals three sources of adaptive variation during a rapid radiation
Pease, J. B., Haak, D. C., Hahn, M. W., & Moyle, L. C. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002379 · Cited by 432 · cited 7×
“A major problem is the intensive computation involved and the inefficient mixing of the transmodel MCMC algorithms used by Bayesian programs, although improvements are being made (e.g., Rannala and Yang 2016).”
BackgroundMethodInfluentialPopulation genomics reveal recent speciation and rapid evolutionary adaptation in polar bears
Liu, S., Lorenzen, E. D., Fumagalli, M., Li, B., Harris, K., Xiong, Z., Zhou, L., Korneliussen, T. S., Somel, M., Babbitt, C., et al. (2014) · Cell · doi:10.1016/j.cell.2014.03.054 · Cited by 411
A rate-independent technique for analysis of nucleic acid sequences: evolutionary parsimony.
(1987) · Molecular Biology and Evolution · doi:10.1093/oxfordjournals.molbev.a040433 · Cited by 377 · cited 1×
“The SVDquartets method for generating the quartet tree is thus similar to the evolutionary parsimony method by Lake (1987), which is also an invariance-based method for quartet data.”
Method
Cited by
Phylogenetic tree building in the genomic age
Kapli, P., Yang, Z., & Telford, M. J. (2020) · Nature Reviews Genetics · doi:10.1038/s41576-020-0233-0 · Cited by 533
Self-citationStarBEAST2 brings faster species tree inference and accurate estimates of substitution rates
Ogilvie, H. A., Bouckaert, R. R., & Drummond, A. J. (2017) · Molecular Biology and Evolution · doi:10.1093/molbev/msx126 · Cited by 469
“In this way very accurate species trees can be estimated using only weakly informative loci, which may not be possible using MP-EST (Xu and Yang 2016).”
BackgroundSpecies Tree Inference with BPP Using Genomic Sequences and the Multispecies Coalescent
Flouri, T., Jiao, X., Rannala, B., & Yang, Z. (2018) · Molecular Biology and Evolution · doi:10.1093/molbev/msy147 · Cited by 445
Self-citationBurbrink, F. T., Grazziotin, F. G., Pyron, R. A., Cundall, D., Donnellan, S., Irish, F., Keogh, J. S., Kraus, F., Murphy, R. W., Noonan, B., et al. (2020) · Systematic Biology · doi:10.1093/sysbio/syz062 · Cited by 317
The Spectre of Too Many Species
Leaché, A. D., Zhu, T., Rannala, B., & Yang, Z. (2019) · Systematic Biology · doi:10.1093/sysbio/syy051 · Cited by 287
Self-citation
Metadata
- Authors
- Xu, B., & Yang, Z.
- Venue
- Genetics
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-12-01
- Volume
- 204
- Issue
- 4
- Pages
- 1353-1368
- ISSN
- 0016-6731 (ISSN-L)
- 1943-2631
- OpenAlex
- W2560579881
- Cited by
- 152
- References
- 12
- FWCI
- 8.73
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Challenges in Species Tree Estimation Under the Multispecies Coalescent Model. https://app.cochat.ai/papers/10.1534/genetics.116.190173
- The paper
- Xu, B., & Yang, Z. (2016). Challenges in Species Tree Estimation Under the Multispecies Coalescent Model. Genetics. https://doi.org/10.1534/genetics.116.190173
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.