Benchmarking Orthogroup Inference Accuracy: Revisiting Orthobench
Open accessPublished version of a preprint
Abstract
Orthobench is the standard benchmark to assess the accuracy of orthogroup inference methods. It contains 70 expert-curated reference orthogroups (RefOGs) that span the Bilateria and cover a range of different challenges for orthogroup inference. Here, we leveraged improvements in tree inference algorithms and computational resources to reinterrogate these RefOGs and carry out an extensive phylogenetic delineation of their composition. This phylogenetic revision altered the membership of 31 of the 70 RefOGs, with 24 subject to extensive revision and 7 that required minor changes. We further used these revised and updated RefOGs to provide an assessment of the orthogroup inference accuracy of widely used orthogroup inference methods. Finally, we provide an open-source benchmarking suite to support the future development and use of the Orthobench benchmark.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The OrthoBench dataset consists of expert-curated reference orthogroups.
“we compared this new version of OrthoFinder with a set of alternative methods on the OrthoBench dataset of expert-curated reference orthogroups20,21.”
From OrthoFinder: improved phylogenetic orthology inference with enhanced accuracy and scalability (2026) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
OrthoFinder: phylogenetic orthology inference for comparative genomics
Emms, D. M., & Kelly, S. (2019) · Genome Biology · doi:10.1186/s13059-019-1832-y · Cited by 7,676 · see citing works
Self-citationCunningham, F., Achuthan, P., Akanni, W., Allen, J., Amode, M. R., Armean, I. M., Bennett, R., Bhai, J., Billis, K., Boddu, S., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1113 · Cited by 925 · see citing works
Flagged first, then by citations
Research works printed but not linked: 30
- Altenhoff AM, et al 2016 Standardized benchmarking in the quest for orthologs. Nat Methods. 13(5): 425– 430. 27043882 PMC4827703
- Altenhoff AM, et al 2020 The Quest for Orthologs benchmark service and consensus calls in 2020. Nucleic Acids Res. 48(W1): W538– W545. 32374845 PMC7319555
- Buchfink B, Xie C, Huson DH. 2015 Fast and sensitive protein alignment using DIAMOND. Nat Methods. 12(1): 59– 60. 25402007
- Camacho C, et al 2009 BLAST+: architecture and applications. BMC Bioinf. 10(1): 421. PMC2803857 20003500
- Capella-Gutierrez S, Silla-Martinez JM, Gabaldon T. 2009 trimAl: a tool for automated alignment trimming in large-scale phylogenetic analyses. Bioinformatics 25(15): 1972– 1973. 19505945 PMC2712344
- Cosentino S, Iwasaki W. 2019 SonicParanoid: fast, accurate and easy orthology inference. Bioinformatics 35(1): 149– 151. 30032301 PMC6298048
- Dessimoz C, et al 2012 Toward community standards in the quest for orthologs. Bioinformatics 28(6): 900– 904. 22332236 PMC3307119
- El-Gebali S, et al 2019 The Pfam protein families database in 2019. Nucleic Acids Res. 47(D1): D427– D432. 30357350 PMC6324024
- Emms DM, Kelly S. 2015 OrthoFinder: solving fundamental biases in whole genome comparisons dramatically improves orthogroup inference accuracy. Genome Biol. 16(1): 157. 26243257 PMC4531804
- Forslund K, et al 2018 Gearing up to handle the mosaic nature of life in the quest for orthologs. Bioinformatics 34(2): 323– 329. 28968857 PMC5860199
- Gabaldon T, et al 2009 Joining forces in the quest for orthologs. Genome Biol. 10(9): 403. 19785718 PMC2768974
- Glover N, et al 2019 Advances and applications in the quest for orthologs. Mol Biol Evol. 36(10): 2157– 2164. 31241141 PMC6759064
- Huerta-Cepas J, Serra F, Bork P. 2016 ETE 3: reconstruction, analysis, and visualization of phylogenomic data. Mol Biol Evol. 33(6): 1635– 1638. 26921390 PMC4868116
- Huson DH, Scornavacca C. 2012 Dendroscope 3: an interactive tool for rooted phylogenetic trees and networks. Syst Biol. 61(6): 1061– 1067. 22780991
- Kaduk M, Sonnhammer E. 2017 Improved orthology inference with Hieranoid 2. Bioinformatics 33(8): 1154– 1159. 28096085
- Kalyaanamoorthy S, Minh BQ, Wong TKF, von Haeseler A, Jermiin LS. 2017 ModelFinder: fast model selection for accurate phylogenetic estimates. Nat Methods. 14(6): 587– 589. 28481363 PMC5453245
- Katoh K, Standley DM. 2013 MAFFT multiple sequence alignment software version 7: improvements in performance and usability. Mol Biol Evol. 30(4): 772– 780. 23329690 PMC3603318
- Larsson A. 2014 AliView: a fast and lightweight alignment viewer and editor for large datasets. Bioinformatics 30(22): 3276– 3278. 25095880 PMC4221126
- Li L, Stoeckert CJ, Roos DS. 2003 OrthoMCL: identification of ortholog groups for eukaryotic genomes. Genome Res. 13(9): 2178– 2189. 12952885 PMC403725
- Mistry J, Finn RD, Eddy SR, Bateman A, Punta M. 2013 Challenges in homology search: HMMER3 and convergent evolution of coiled-coil regions. Nucleic Acids Res. 41(12): e121. 23598997 PMC3695513
- Muller J, Creevey CJ, Thompson JD, Arendt D, Bork P. 2010 AQUA: automated quality improvement for multiple sequence alignments. Bioinformatics 26(2): 263– 265. 19926669
- Muller J, et al 2005 Sequence and comparative genomic analysis of actin-related proteins. Mol Biol Cell 16(12): 5736– 5748. 16195354 PMC1289417
- Nguyen LT, Schmidt HA, von Haeseler A, Minh BQ. 2015 IQ-TREE: a fast and effective stochastic algorithm for estimating maximum-likelihood phylogenies. Mol Biol Evol. 32(1): 268– 274. 25371430 PMC4271533
- Sievers F, Higgins DG. 2020 QuanTest2: benchmarking multiple sequence alignments using secondary structure prediction. Bioinformatics 36: 90– 95. 31292629 PMC9881607
- Sonnhammer EL, et al 2014 Big data and other challenges in the quest for orthologs. Bioinformatics 30(21): 2993– 2998. 25064571 PMC4201156
- Stamatakis A. 2014 RAxML version 8: a tool for phylogenetic analysis and post-analysis of large phylogenies. Bioinformatics 30(9): 1312– 1313. 24451623 PMC3998144
- Thompson JD, Plewniak F, Ripp R, Thierry JC, Poch O. 2001 Towards a reliable objective function for multiple sequence alignments. J Mol Biol. 314(4): 937– 951. 11734009
- Trachana K, et al 2011 Orthology prediction methods: a quality assessment using curated protein families. BioEssays 33(10): 769– 780. 21853451 PMC3193375
- Trachana K, et al 2014 A phylogeny-based benchmarking test for orthology inference reveals the limitations of function-based validation. PLoS One 9(11): e111122. 25369365 PMC4219706
- Train CM, Glover NM, Gonnet GH, Altenhoff AM, Dessimoz C. 2017 Orthologous Matrix (OMA) algorithm 2.0: more robust to asymmetric evolutionary rates and more scalable hierarchical orthologous group inference. Bioinformatics 33(14): i75– i82. 28881964 PMC5870696
Cited by
Sorted by citations
OrthoFinder: improved phylogenetic orthology inference with enhanced accuracy and scalability
Emms, D. M., Liu, Y., Belcher, L., Holmes, J., & Kelly, S. (2026) · Nature Methods · doi:10.1038/s41592-026-03126-6 · Cited by 25 · see citing works
Self-citation
- Authors
- Emms, D. M., & Kelly, S.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Emms, David M
- Kelly, Steven
- Venue
- Genome Biology and Evolution
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-10-06
- Volume
- 12
- Issue
- 12
- Pages
- 2258-2266
- ISSN
- 1759-6653 (ISSN-L)
- PMC
- PMC7738749
- OpenAlex
- W3092516389
- Cited by
- 36
- References
- 2
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 7 · 2023: 2 · 2022: 11 · 2021: 5
- FWCI
- 1.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Genome Biology and Evolution (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Oxford University Research Archive (ORA) (University of Oxford) (submittedVersion, cc-by)
- Funders
- European Union’s Horizon 2020 Research and Innovation Programme (637765)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Agricultural and Biological Sciences › Plant Science › Chromosomal and Genetic Variations
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- benchmarking, Bilateria, benchmark suite
- MeSH terms
- Biological Evolution, Genetic Techniques, Databases, Factual, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
