The unsolved challenge to phylogenetic correlation tests for categorical characters

Maddison and FitzJohn (2015) simulated comparative data and reported a preponderance of significant results using Pagel's correlation test (1994) and Maddison's (1990) concentrated changes test.

“In their article, Maddison and FitzJohn (2015) simulated comparative data and reported a preponderance of significant results using Pagel's correlation test (1994) and Maddison's (1990) concentrated changes test.”

Common statistical tests for the evolutionary correlation of discrete characters are prone to reporting a significant association even when the pattern is driven by a single or very few independent transitions.

“Maddison and FitzJohn (2015) recently demonstrated that common statistical tests (e.g., Maddison 1990;Pagel 1994) for the evolutionary correlation of discrete characters are prone to reporting a significant association even when the pattern is driven by a single (or, very few) independent transition(s) from one character state to another.”

These scenarios ultimately reduce to whether or not coincident unreplicated events are evidence of a causal link between traits.

“As with Case Study I, these scenarios ultimately reduce to whether or not coincident unreplicated events are evidence of a causal link between traits (Maddison and FitzJohn 2015), a problem we will address in Case Study III.”

Maddison and FitzJohn (2015) demonstrated that commonly used methods return significant correlations all the time, in scenarios that defy statistical intuition.

“As we mention above, Maddison and FitzJohn (2015) recently demonstrated that commonly used methods return significant correlations all the time—and in scenarios that seem to defy our statistical intuition.”

This is a clear case of phylogenetic pseudoreplication.

“This seems to be a clear case of phylogenetic pseudoreplication (Read and Nee 1995;Maddison and FitzJohn 2015).”

The authors ultimately disagree with Maddison and FitzJohn (2015) that finding whether two variables are part of the same adaptive/functional network constitutes a weak conclusion; the challenges of making these inferences from any comparative data set are significant.

“We ultimately disagree with them that this constitutes a weak conclusion; the challenges of making these inferences from any comparative data set are significant.”

Maddison and FitzJohn (2015) hinted that the coincident occurrence of single events could be a way of measuring the evidence for a correlation, but did not work out the details as the authors have done here.

“Maddison and FitzJohn (2015) hinted that the coincident occurrence of single events could be a way of measuring the evidence for a correlation, but did not work out the details as we have done here.”

The traits used in comparative analyses are not chosen independently with respect to their phylogenetic distribution.

“As Maddison and FitzJohn (2015) point out, the traits we use in comparative analyses are not chosen independently with respect to their phylogenetic distribution (as we assumed in our analysis).”

The authors agree with Maddison and FitzJohn (2015) that ascertainment bias is likely prevalent in empirical studies.

“We agree with Maddison and FitzJohn (2015) that this type of ascertainment bias is likely prevalent in empirical studies, even if it is usually more subtle than testing for a correlation between milk and middle ear bones.”

The level of significance (the size of a P-value) will depend on the size of the effect and the number of data points, tested against a null hypothesis of exactly zero difference, or exactly zero correlation.

“Usually, the level of significance (the size of a P-value) will depend on the size of the effect and the number of data points, tested against a null hypothesis of exactly zero difference, or exactly zero correlation119–121.”