CoChat
Sign Up
PreprintCited by 53Genomics and Phylogenetic StudiesRNA and protein synthesis mechanismsEvolution and Paleontology Studies

Six-state amino acid recoding is not an effective strategy to offset the effects of compositional heterogeneity and saturation in phylogenetic analyses

Open access
Hernandez, Alexandra M., Ryan, Joseph F.·2019·Systematic Biology
DOI 10.1101/729103PMC PMC8513762

Preprint — published as 10.1093/sysbio/syab027 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Six-state amino acid recoding strategies are commonly applied to combat the effects of compositional heterogeneity and substitution saturation in phylogenetic analyses. While these methods have been endorsed from a theoretical perspective, their performance has never been extensively tested. Here, we test the effectiveness of 6-state recoding approaches by comparing the performance of analyses on recoded and non-recoded datasets that have been simulated under gradients of compositional heterogeneity or saturation. In all of our simulation analyses, non-recoding approaches greatly outperformed 6-state recoding approaches. Our results suggest that 6-state recoding strategies are not effective in the face of high saturation. Further, while recoding strategies do buffer the effects of compositional heterogeneity, the loss of information that accompanies 6-state recoding outweighs its benefits, even in the most compositionally heterogeneous datasets. In addition, we evaluate recoding schemes with 9, 12, 15, and 18 states and show that these all outperform 6-state recoding. Our results have important implications for the more than 70 published papers that have incorporated 6-state recoding, many of which have significant bearing on relationships across the tree of life.

AI snapshot

Written from the paper’s full text.

TL;DR

Six-state amino acid recoding is commonly used to counter compositional heterogeneity and saturation, but simulations show it generally performs worse than non-recoding for phylogenetic reconstruction. Recoding with more states (9–18) can outperform six-state recoding, yet benefits are limited and recoding should be justified rather than applied routinely.

Key findings

  • Across simulated gradients of compositional heterogeneity and saturation, non-recoded analyses consistently produced more accurate trees than six-state recoded analyses; for saturation, Robinson-Foulds distances were significantly higher for recoded data sets (P < 2.2e-16).
  • Six-state recoding buffered the effects of compositional heterogeneity, but the loss of information outweighed its benefits; non-recoding performed significantly better in all tested cases except under the highest compositional heterogeneity and shortest stem branch.
  • Dayhoff 9-, 12-, 15-, and 18-state recoding generally outperformed six-state recoding; Dayhoff 18-state recoding outperformed all strategies including non-recoding in one analysis, but its difference from non-recoding was significant only under the highest compositional heterogeneity (P = 6.622e-06; other P-values 0.4314 and 0.2183).
  • At least 91 published phylogenetic studies have used six-state amino acid recoding; simulated data with inflation parameter 0.9 had a median RCFV of 0.088 versus 0.036 for real data from the surveyed papers, and RCFV ≥ 0.1 occurred in 6 out of 25 sampled publications.

Methods

  • The study used empirical data from Chang et al. (2015) and Feuda et al. (2017), comprising 51,940 amino acid positions from 78 taxa plus 9 non-animal outgroups, alongside simulated data under gradients of compositional heterogeneity or saturation.
  • Compositional heterogeneity was simulated in P4 on four hypothetical 20-taxon balanced trees with 1000-amino-acid sequences and inflation parameters 0.1, 0.5, and 0.9; data were recoded with Dayhoff and S&R six-state schemes, analyzed in RAxML, and evaluated with Robinson-Foulds distances and a deep-splits metric using chi-squared tests with Bonferroni correction at α = 0.002.
  • Saturation was simulated in Seq-Gen on the Chang and Feuda trees with branch-length scaling factors 1–20 and Dayhoff/JTT models, producing 80,000 data sets plus 5,000 GTR data sets (85,000 total) and 180,000 phylogenies; Robinson-Foulds distances were compared with t-tests and Bonferroni correction at α = 0.0009. Alternative Dayhoff 9-, 12-, 15-, and 18-state binning schemes were also developed and tested under compositional heterogeneity.

Limitations

  • Stated: the authors caution against blindly applying Dayhoff 18-state recoding because significant improvement over non-recoding occurred only under the most extreme compositional heterogeneity setting (0.9), which they show is uncommon in real data; they also state that applying any recoding should include sufficient justification and that more analyses are needed to understand alternative recoding methods.
  • Stated/evident: alternative binning schemes were not identified exhaustively because the number of possible bins is computationally intractable; the authors instead chose best-scoring schemes from subsets of Dayhoff six-state bins and compared them to Susko and Roger’s schemes.
  • Evident: the conclusions are based on simulations using specific models, trees, and empirical data sets from Chang and Feuda, so performance may not generalize to all real data sets or to recoding schemes not tested here.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/729103

Cites

135 of 135 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 19 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Recoding strategies buffer the effects of compositional heterogeneity.

    “while recoding strategies do buffer the effects of compositional heterogeneity”
    Quote in the paperSubstantial claim
  • Findingown data

    In simulation analyses, non-recoding approaches consistently outperform six-state recoding approaches.

    “In our simulation analyses, non-recoding approaches consistently outperform six-state recoding approaches.”
    Quote in the paperSubstantial claim
  • Findingown data

    Recoding schemes with 9, 12, 15, and 18 states consistently outperform six-state recoding.

    “In addition, we evaluate recoding schemes with 9, 12, 15, and 18 states and show that these consistently outperform six-state recoding.”
    Quote in the paperSubstantial claim
  • Findingown data

    Despite changes to stem branch length and level of compositional heterogeneity, non-recoding methods produced more accurate trees.

    “non-recoding methods produced more accurate trees”
    Quote in the paperSubstantial claim
  • Findingown data

    Significant improvement only occurs under the most extreme compositional heterogeneity setting (0.9).

    “significant improvement only occurs under the most extreme compositional heterogeneity setting (0.9)”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 5
  • Classified as probably not a substantial claim. 15
  • May combine two claims. 12

  • The quote only partly states this claim.

    Findingown data

    T-tests performed for each branch length scaling factor parameter showed that Robinson-Foulds distances were significantly higher for recoded data sets compared to non-recoded data sets (P-value < 2.2e-16).

    “Robinson–Foulds distances were significantly higher for recoded data sets compared to non-recoded data sets (P-value < 2.2e-16)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    There are at least 91 publications that use six-state amino acid recoding, with 2019 seeing more than any year to date.

    “There are at least 91 publications that use six-state amino acid recoding, with 2019 seeing more than any year to date”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    At least 91 phylogenetic studies have implemented six-state amino acid recoding strategies.

    “To date, there are at least 91 phylogenetic studies that have implemented six-state amino acid recoding strategies, with the highest number of studies published in 2019 (Table 1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The median RCFV score was 0.088 for all data sets simulated under the inflation parameter of 0.9, while the median RCFV score for data from papers in Table 1 was 0.036.

    “The median RCFV score was 0.088 for all data sets simulated under the inflation parameter of 0.9, while the median RCFV score for data from papers in Table 1 was 0.036”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The percentage of trees that passed the deep splits criteria increased as the number of Dayhoff states increased, with Dayhoff 18-state recoding outperforming all other strategies including the non-recoding approach.

    “The percentage of trees that passed these criteria increased as the number of Dayhoff states increased with Dayhoff 18-state recoding outperforming all other strategies including the non-recoding approach”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Increases in sequence length minimized the impact of saturation and reduced errors in phylogenetic reconstruction for both recoding and non-recoding methods.

    “Increases in sequence length minimized the impact of saturation and reduced errors in phylogenetic reconstruction for both recoding and non-recoding methods”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Analyses of non-recoded data sets consistently produced trees that were more accurate than those produced on recoded data sets using both deep splits metric and Robinson-Foulds distances.

    “Analyses of non-recoded data sets consistently produced trees that were more accurate than those produced on recoded data sets”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Non-recoding methods performed significantly better on all sequence lengths and levels of saturation, except for on the largest simulated data set with the lowest level of saturation where results from recoded and non-recoded analyses were equivocal.

    “non-recoding methods performed significantly better on all sequence lengths and levels of saturation, except for on the largest simulated data set with the lowest level of saturation where results from recoded and non-recoded analyses were equivocal”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The results were consistent regardless of which tree (Chang or Feuda) was used for data simulations.

    “The results were consistent regardless of which tree (i.e., Chang or Feuda) was used for data simulations”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In all cases, the intra-bin substitution scores generated by the authors' best binning strategies were higher than those proposed by Susko and Roger (2007), except for one which had an equal score.

    “in all cases, the scores we generated were higher, except for one which had an equal score”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Susko and Roger (2007) source

      Not checked in the free run.

  • The quote only partly states this claim.

    Findingown data

    The difference in numbers of incorrect trees between analyses run with Dayhoff 18-state recoding and those run without recoding was significant only under the highest level of compositional heterogeneity (P-values for inflation parameters 0.1, 0.5, and 0.9: 0.4314, 0.2183, and 6.622e-06, respectively).

    “The difference was significant only under the highest level of compositional heterogeneity”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Six-state recoding methods have been endorsed from a theoretical perspective, but their performance has never been extensively tested.

    “While these methods have been endorsed from a theoretical perspective, their performance has never been extensively tested.”
    • Also stated in: another passage · plain
      “While amino acid recoding has been considered from a theoretical perspective (Davidson et al. 2002; Embley et al. 2003a; Hrdy et al. 2004), and there have been comparisons between different recoding strategies (Susko and Roger 2007), there has not been extensive empirical testing of the widely applied six-state recoding approaches.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Davidson et al. 2002

      Not checked in the free run.

    • cites Embley et al. 2003a

      Not checked in the free run.

    • cites Hrdy et al. 2004 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is a large body of research on the conditions for state aggregation (or lumpability) in modeling character data such as DNA or amino acids.

    “There is a large body of research on the conditions for state aggregation (or lumpability) in modeling character data such as DNA or amino acids (Kemeny and Snell 1976; Courtois 1977).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Kemeny and Snell 1976

      Not checked in the free run.

    • cites Courtois 1977

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Six-state amino acid recoding strategies are commonly applied to combat the effects of compositional heterogeneity and substitution saturation in phylogenetic analyses.

    “Six-state amino acid recoding strategies are commonly applied to combat the effects of compositional heterogeneity and substitution saturation in phylogenetic analyses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Compositional heterogeneity and substitution saturation are major challenges to phylogenetic inference.

    “Compositional heterogeneity and substitution saturation are major challenges to phylogenetic inference.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Collins et al. 1994

      Not checked in the free run.

    • cites Foster and Hickey 1999

      Not checked in the free run.

    • cites Ho and Jermiin 2004

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Matrix recoding has been proposed as a solution for both compositional heterogeneity and substitution saturation.

    “Based on this foundation, matrix recoding has been proposed as a solution for both compositional heterogeneity and substitution saturation (Blanquart and Lartillot 2006; Susko and Roger 2007).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Blanquart and Lartillot 2006

      Not checked in the free run.

    • cites Susko and Roger 2007 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    To correct for multiple t-tests, the authors applied a Bonferroni correction with alpha = 0.0009.

    “we applied the Bonferroni correction at which”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationreasoning

    The authors' simulated data sets are substantially compositionally heterogeneous because published data sets, many of which used compositional heterogeneity as justification for recoding, are likely enriched for compositional heterogeneity.

    “We reason that our simulated data sets therefore are substantially compositionally heterogeneous”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The published data sets in Table 1 are enriched for compositional heterogeneity.

  • May combine two claims.

    Premisecitation

    The number of possible amino acid binning schemes for a given number of states is very large, making an exhaustive search computationally intractable.

    “the number of possible bins is very large”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Susko and Roger (2007) source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The deep splits metric focuses on errors most likely due to convergent amino acid compositions.

    “The rationale for this metric rather than Robinson–Foulds distances is that it focused on errors that were most likely due to convergent amino acid compositions (i.e., the pulling together of compositionally homogeneous but unrelated clades or tips).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    All analyses were pre-planned using phylotocol and pre-registered using the Center for Open Science's pre-registration platform.

    “To maximize transparency and minimize confirmation bias, all analyses were pre-planned using phylotocol (DeBiasse and Ryan 2019) and pre-registered using the Center for Open Science’s pre-registration platform (https://osf.io/smj6k/ and https://osf.io/6ubgj/).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites DeBiasse and Ryan 2019

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Prior to initial submission, four changes were made to the original phylotocol plan.

    “Prior to the initial submission of this manuscript, we made four changes to the original plan outlined in our phylotocol.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Feuda extensively applied six-state amino acid recoding to the Chang data set in a reanalysis.

    “Feuda extensively applied six-state amino acid recoding to this data set in a reanalysis of the Chang study, which did not use recoding.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Feuda et al. 2017 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The topologies from Chang and Feuda are based on the same data set made up of 51,940 amino acid positions from 78 taxa representing a wide range of animals and 9 non-animal outgroups.

    “The topologies from Chang and Feuda are based on the same data set which is made up of 51,940 amino acid positions from 78 taxa representing a wide range of animals and 9 non-animal outgroups.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chang et al. 2015 source

      Not checked in the free run.

    • cites Feuda et al. 2017 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Six-state recoding greatly reduces information content and therefore often results in suboptimal phylogenetic reconstructions.

    “six-state recoding greatly reduces information content and therefore often results in suboptimal phylogenetic reconstructions”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Study design and conductnone given

    To correct for multiple chi-squared tests, the authors applied a Bonferroni correction with alpha = 0.017.

    “we applied the Bonferroni correction at which”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Limitations extend beyond Dayhoff matrices, as six-states likely are too few for reliable phylogenetic analysis.

    “limitations extend beyond Dayhoff matrices, as six-states likely are too few for reliable phylogenetic analysis”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Historically, simulation has been an effective strategy for empirically testing the performance of phylogenetic approaches.

    “Historically, simulation has been an effective strategy for empirically testing the performance of phylogenetic approaches (Kuhner and Felsenstein 1994; Swofford et al. 2001; Zwickl et al. 2002; Kubatko and Degnan 2007; Huang and Knowles 2016).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Kuhner and Felsenstein 1994

      Not checked in the free run.

    • cites Swofford et al. 2001

      Not checked in the free run.

    • cites Zwickl et al. 2002 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    While six-state recoding seems to be less affected by increases in compositional heterogeneity, it does not overcome the penalty of information loss even under the highest levels of compositional heterogeneity.

    “while six-state recoding seems to be less affected by increases in compositional heterogeneity, it does not overcome the penalty of information loss even under the highest levels of compositional heterogeneity”
    Quote in the paperSubstantial claim
29 of 29

  • The paper relies on this without support.

    The tree topologies and branch lengths used in the simulations are representative of real phylogenetic data sets.

    “We simulated data sets on the Chang and Feuda trees under the Dayhoff and JTT models with increasing levels of saturation.”

  • The paper relies on this without support.

    The published data sets in Table 1 are enriched for compositional heterogeneity.

    “many of which used compositional heterogeneity as justification for the application of recoding, are likely enriched for compositional heterogeneity”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported amino-acid pairing repeats F and omits C

    Minor

    The stated procedure pairs the first ten amino acids with the second ten. The reported pairs instead use F twice, omit C, and place F on both sides of the pairing.

  • One Luo et al. (2014) citation has two organismal scopes

    Minor

    Table 1 lists the same citation once for bivalves and again for Roseobacter. The reference list identifies Luo et al. (2014) as the Roseobacter study, leaving the bivalve entry inconsistent.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Hernandez, A. M., & Ryan, J. F.
Affiliations
University of Florida (US)
Corresponding author
Ryan, Joseph F.
ORCID
Ryan, Joseph F.
Venue
Systematic Biology
Publisher
openRxiv
Published
2019-08-08
Volume
70
Issue
6
Pages
1200-1212
ISSN
1063-5157 (ISSN-L)
DOI
10.1101/729103
PMC
PMC8513762
OpenAlex
W3143686523
Cited by
52
References
134
Citations by year
2026: 6 · 2025: 5 · 2024: 11 · 2023: 11 · 2022: 11 · 2021: 3 · 2020: 4 · 2019: 1
FWCI
2.82
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Systematic Biology (publishedVersion, cc-by-nc)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DRYAD (public-domain)
Funders
  • National Science Foundation (1542597)
  • McKnight Foundation
  • Florida Education Fund
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
  • Physical Sciences › Earth and Planetary Sciences › Paleontology › Evolution and Paleontology Studies
Keywords
compositional heterogeneity, substitution saturation, phylogenetic analysis, information loss, simulation study, protein sequence data
MeSH terms
Amino Acids, Phylogeny

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review