CoChat
Sign Up
Cited by 242Genomics and Phylogenetic StudiesFractal and DNA sequence analysisProtein Structure and Dynamics

A Comprehensive Benchmark Study of Multiple Sequence Alignment Methods: Current Challenges and Future Perspectives

Open access
Thompson, Julie D., Linard, Benjamin, Lecompte, Odile, Poch, Olivier·2011·PLoS ONE
DOI 10.1371/journal.pone.0018093PMC PMC3069049
Open this paper in CoChat — free account

Abstract

Multiple comparison or alignmentof protein sequences has become a fundamental tool in many different domains in modern molecular biology, from evolutionary studies to prediction of 2D/3D structure, molecular function and inter-molecular interactions etc. By placing the sequence in the framework of the overall family, multiple alignments can be used to identify conserved features and to highlight differences or specificities. In this paper, we describe a comprehensive evaluation of many of the most popular methods for multiple sequence alignment (MSA), based on a new benchmark test set. The benchmark is designed to represent typical problems encountered when aligning the large protein sequence sets that result from today's high throughput biotechnologies. We show that alignmentmethods have significantly progressed and can now identify most of the shared sequence features that determine the broad molecular function(s) of a protein family, even for divergent sequences. However,we have identified a number of important challenges. First, the locally conserved regions, that reflect functional specificities or that modulate a protein's function in a given cellular context,are less well aligned. Second, motifs in natively disordered regions are often misaligned. Third, the badly predicted or fragmentary protein sequences, which make up a large proportion of today's databases, lead to a significant number of alignment errors. Based on this study, we demonstrate that the existing MSA methods can be exploited in combination to improve alignment accuracy, although novel approaches will still be needed to fully explore the most difficult regions. We then propose knowledge-enabled, dynamic solutions that will hopefully pave the way to enhanced alignment construction and exploitation in future evolutionary systems biology studies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0018093

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current research in multiple sequence alignment has focused on improving alignment accuracy, improving scalability to handle large datasets, and incorporating evolutionary models and structural information into the alignment process.

    “Current research in MSA has focused on improving alignment accuracy, improving scalability to handle large datasets, and incorporating evolutionary models and structural information into the alignment process [26].”

    From Whole-Genome Alignment: Methods, Challenges, and Future Directions (2024) · cites this paper as [26]

    The Introduction describes efforts to improve alignment accuracy and identifies scalability as a challenge for large sequence sets. The Discussion says structural information has been incorporated into alignment methods. The paper also describes phylogenetic-tree-guided alignment, but the cited passages do not establish incorporating evolutionary models as a current research focus. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Important challenges regarding multiple sequence alignment include computational expense, accurate alignment, and selecting a gap penalty that accurately reflects biological reality.

    “However, there are a number of important challenges regarding this algorithm, such as computationally expensive, accurate alignment and selecting a gap penalty that accurately reflects biological reality [26].”

    From Whole-Genome Alignment: Methods, Challenges, and Future Directions (2024) · cites this paper as [26]

    The Abstract (page unavailable) identifies alignment errors and difficulty aligning locally conserved or disordered regions; the Introduction (page unavailable) also notes scalability issues with growing sequence datasets. These support accuracy and computational-scale challenges, but the passages do not identify choosing a biologically realistic gap penalty as a challenge. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • More accurate multiple sequence alignment and tree inference methods should give more accurate ortholog inference.

    “More accurate multiple sequence alignment and tree inference methods should give more accurate ortholog inference, and many studies exist comparing the accuracy and runtime characteristics of the available methods [36, 37].”

    From OrthoFinder: phylogenetic orthology inference for comparative genomics (2019) · cites this paper as [36]

    The Results section ('MSA program evaluation: overall alignment quality'; page not given) compares alignment accuracy and computation time across methods. The Introduction describes MSAs as prerequisites for evolutionary analysis, but our copy does not establish that more accurate alignment and tree-inference methods yield more accurate ortholog inference. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Broad phylogenomic sampling improves resolution of the animal tree of life

    Dunn, C. W., Hejnol, A., Matus, D. Q., Pang, K., Browne, W. E., Smith, S. A., Seaver, E., Rouse, G. W., Obst, M., Edgecombe, G. D., et al. (2008) · Nature · doi:10.1038/nature06614 · Cited by 1,921 · cited 1× · see citing works

    “Thus, evolutionary-based inference systems are playing an increasingly important role in diverse areas, such as elucidation of the tree of life [2], studies of epidemiology and virulence [3], drug design [4], human genetics [5], cancer [6] or biodiversity [7].”
    Background

Flagged first, then by citations

Research works printed but not linked: 71

  1. [1]Harvey PH Pagel MD 1991 The Comparative Method in Evolutionary Biology. Oxford University Press Paris
  2. [3]Bao Y Bolotov P Dernovoy D Kiryutin B Zaslavsky L 2008 The influenza virus resource at the National Center for Biotechnology Information. J Virol 82 596 601 17942553 PMC2224563
  3. [4]Kuipers RK Joosten HJ van Berkel WJ Leferink NG Rooijen E 2010 3DM: systematic analysis of heterogeneous superfamily data to discover protein functionalities. Proteins 78 2101 2113 20455266
  4. [5]Singh S Tokhunts R Baubet V Goetz JA Huang ZJ 2009 Sonic hedgehog mutations identified in holoprosencephaly patients can act in a dominant negative manner. Hum Genet 125 95 103 19057928 PMC2692056
  5. [6]Zhang J Chen X Kent M Rodriguez C Chen X 2009 Establishment of a dog model for the p53 family pathway and identification of a novel isoform of p21 cyclin-dependent kinase inhibitor. Mol Cancer Res 7 67 78 19147538 PMC2653090
  6. [7]Eaton MJ Martin A Thorbjarnarson J Amato G 2009 Species-level diversification of African dwarf crocodiles (Genus Osteolaemus): a geographic and phylogenetic perspective. Mol Phylogenet Evol 50 496 506 19056500
  7. [8]Levasseur A Pontarotti P Poch O Thompson JD 2008 Strategies for reliable exploitation of evolutionary concepts in high throughput biology. Evol Bioinform Online 4 121 137 19204813 PMC2614184
  8. [9]Wong KM Suchard MA Huelsenbeck JP 2008 Alignment uncertainty and genomic analysis. Science 319 473 476 18218900
  9. [10]Löytynoja A Goldman N 2008 Phylogeny-aware gap placement prevents errors in sequence alignment and evolutionary analysis. Science 320 1632 1635 18566285
  10. [11]Brown DP Krishnamurthy N Sjolander K 2007 Automated protein subfamily identification and classification. PLoS Comput Biol 3 e160 17708678 PMC1950344
  11. [12]Brandt BW Feenstra KA Heringa J 2010 Multi-Harmony: detecting functional specificity from sequence alignment. Nucleic Acids Res 38 Suppl W35 40 20525785 PMC2896201
  12. [13]Rausell A Juan D Pazos F Valencia A 2010 Protein interactions and ligand binding: from protein subfamilies to functional specificity. Proc Natl Acad Sci U S A 107 1995 2000 20133844 PMC2808218
  13. [14]Feng DF Doolittle RF 1987 Progressive sequence alignment as a prerequisite to correct phylogenetic trees. J Mol Evol 25 351 360 3118049
  14. [15]Thompson JD Plewniak F Poch O 1999 BAliBASE: a benchmark alignment database for the evaluation of multiple alignment programs. Bioinformatics 15 87 88 10068696
  15. [16]Gardner PP Wilm A Washietl S 2005 A benchmark of multiple sequence alignment programs upon structural RNAs. Nucleic Acids Res 33 2433 2439 15860779 PMC1087786
  16. [17]Gotoh O 1996 Significant improvement in accuracy of multiple protein sequence alignments by iterative refinement as assessed by reference to structural alignments. J Mol Biol 264 823 838 8980688
  17. [18]Eddy S 1998 Profile hidden Markov models. Bioinformatics 14 755 763 9918945
  18. [19]Notredame C Higgins DG 1996 SAGA: sequence alignment by genetic algorithm. Nucleic Acids Res 24 1515 1524 8628686 PMC145823
  19. [20]Thompson JD Plewniak F Poch O 1999 A comprehensive comparison of multiple sequence alignment programs. Nucleic Acids Res 27 2682 2690 10373585 PMC148477
  20. [21]Blackshields G Wallace IM Larkin M Higgins DG 2006 Analysis and comparison of benchmarks for multiple sequence alignment. In Silico Biol 6 321 339 16922695
  21. [22]Wallace IM O'Sullivan O Higgins DG Notredame C 2006 M-Coffee: combining multiple sequence alignment methods with T-Coffee. Nucleic Acids Res 34 1692 1699 16556910 PMC1410914
  22. [23]Katoh K Toh H 2008 Recent developments in the MAFFT multiple sequence alignment program. Brief Bioinform 9 286 298 18372315
  23. [24]Edgar RC 2004 MUSCLE: multiple sequence alignment with high accuracy and high throughput. Nucleic Acids Res 32 1792 1797 15034147 PMC390337
  24. [25]Do CB Mahabhashyam MS Brudno M Batzoglou S 2005 ProbCons: Probabilistic consistency-based multiple sequence alignment. Genome Res 15 330 340 15687296 PMC546535
  25. [26]O'Sullivan O Suhre K Abergel C Higgins DG Notredame C 2004 3DCoffee: combining protein sequences and structures within multiple sequence alignments. J Mol Biol 340 385 395 15201059
  26. [27]Chakrabarti S Lanczycki C Panchenko A Przytycka T Thiessen P 2006 Refining multiple sequence alignments with conserved core regions. Nucleic Acids Res 34 2598 2606 16707662 PMC1463900
  27. [28]Lee C Grasso C Sharlow MF 2002 Multiple sequence alignment using partial order graphs. Bioinformatics 18 452 464 11934745
  28. [29]Raghava GP Searle SM Audley PC Barber JD Barton GJ 2003 OXBench: a benchmark for evaluation of protein multiple sequence alignment accuracy. BMC Bioinformatics 4 47 14552658 PMC280650
  29. [30]Dessimoz C Gil M 2010 Phylogenetic assessment of alignments reveals neglected tree signal in gaps. Genome Biol 11 R37 20370897 PMC2884540
  30. [31]Aniba MR Poch O Thompson JD 2010 Issues in bioinformatics benchmarking: the case study of multiple sequence alignment. Nucleic Acids Res 38 7353 7363 20639539 PMC2995051
  31. [32]Koonin EV 2009 Darwinian evolution in the light of genomics. Nucleic Acids Res 37 1011 1034 19213802 PMC2651812
  32. [33]Bakke P Carney N Deloache W Gearing M Ingvorsen K 2009 Evaluation of three automated genome annotations for Halorhabdus utahensis. PLoS One 4 e6291 19617911 PMC2707008
  33. [34]Keller O Odronitz F Stanke M Kollmar M Waack S 2008 Scipio: using protein sequences to determine the precise exon/intron structures of genes and their orthologs in closely related species. BMC Bioinformatics 9 278 18554390 PMC2442105
  34. [35]Guigo R Flicek P Abril JF Reymond A Lagarde J 2006 EGASP: the human ENCODE Genome Annotation Assessment Project. Genome Biol 7 Suppl 1: S2 1-31 PMC1810551 16925836
  35. [36]Mardis ER 2008 The impact of next-generation sequencing technology on genetics. Trends Genet 24 133 141 18262675
  36. [37]Pop M Salzberg SL 2008 Bioinformatics challenges of new sequencing technology. Trends Genet 24 142 149 18262676 PMC2680276
  37. [38]Dunker AK Oldfield CJ Meng J Romero P Yang JY 2008 The unfoldomics decade: an update on intrinsically disordered proteins. BMC Genomics 9 S1 PMC2559873 18831774
  38. [39]Wong WC Maurer-Stroh S Eisenhaber F 2010 More than 1,001 problems with protein domain databases: transmembrane regions, signal peptides and the issue of sequence homology. PLoS Comput Biol 6 e1000867 20686689 PMC2912341
  39. [40]Thompson JD Plewniak F Ripp R Thierry JC Poch O 2001 Towards a reliable objective function for multiple sequence alignments. J Mol Biol 4 937 951 11734009
  40. [41]Bianchetti L Thompson JD Lecompte O Plewniak F Poch O 2005 vALId: validation of protein sequence quality based on multiple alignment data. J Bioinform Comput Biol 3 929 947 16078368
  41. [42]Krause L Diaz NN Bartels D Edwards RA Pühler A 2006 Finding novel genes in bacterial communities isolated from the environment. Bioinformatics 22 e281 289 16873483
  42. [43]Huson DH Auch AF Qi J Schuster SC 2007 MEGAN analysis of metagenomic data. Genome Res 17 377 286 17255551 PMC1800929
  43. [44]Chica C Labarga A Gould CM López R Gibson TJ 2008 A tree-based conservation scoring method for short linear motifs in multiple alignments of protein sequences. BMC Bioinformatics 9 229 18460207 PMC2396637
  44. [45]Sankararaman S Sjölander K 2008 INTREPID—INformation-theoretic TREe traversal for Protein functional site Identification. Bioinformatics 24 2445 2452 18776193 PMC2572704
  45. [46]Amaral PP Dinger ME Mercer TR Mattick JS 2008 The eukaryotic genome as an RNA machine. Science 319 1787 1789 18369136
  46. [48]Simossis V Heringa J 2005 PRALINE: a multiple sequence alignment toolbox that integrates homology-extended and secondary structure information. Nucleic Acids Res 33 W289 294 15980472 PMC1160151
  47. [49]Pei J Grishin N 2007 PROMALS: towards accurate multiple sequence alignments of distantly related proteins. Bioinformatics 23 802 808 17267437
  48. [50]Thompson JD Koehl P Ripp R Poch O 2005 BAliBASE 3.0: latest developments of the multiple sequence alignment benchmark. Proteins 61 127 136 16044462
  49. [51]Schäffer A Aravind L Madden T Shavirin S Spouge J 2001 Improving the accuracy of PSI-BLAST protein database searches with composition-based statistics and other refinements. Nucleic Acids Res 29 2994 3005 11452024 PMC55814
  50. [52]The UniProt Consortium 2009 The Universal Protein Resource (UniProt) in 2010. Nucleic Acids Res. In press PMC1669721 17142230
  51. [53]Berman HM 2008 The Protein Data Bank: a historical perspective. Acta Cryst A64 88 95 18156675
  52. [54]Taylor WR 2000 Protein structure comparison using SAP. Methods Mol Biol 143 19 32 11084900
  53. [55]Plewniak F Thompson JD Poch O 2000 Ballast: blast post-processing based on locally conserved segments. Bioinformatics 16 750 759 11108697
  54. [56]Thompson J Plewniak F Thierry J Poch O 2000 DbClustal: rapid and reliable global multiple alignments of protein sequences detected by database searches. Nucleic Acids Res 28 2919 2926 10908355 PMC102675
  55. [57]Thompson JD Prigent V Poch O 2004 LEON: multiple aLignment Evaluation Of Neighbours. Nucleic Acids Res 32 1298 1307 14982955 PMC390283
  56. [58]Thompson JD Muller A Waterhouse A Procter J Barton GJ 2006 MACSIMS: multiple alignment of complete sequences information management system. BMC Bioinformatics 7 318 16792820 PMC1539025
  57. [59]Waterhouse AM Procter JB Martin DM Clamp M Barton GJ 2009 Jalview Version 2—a multiple sequence alignment editor and analysis workbench. Bioinformatics 25 1189 1191 19151095 PMC2672624
  58. [60]Thompson JD Thierry JC Poch O 2003 RASCAL: rapid scanning and correction of multiple sequence alignments. Bioinformatics 19 1155 1161 12801878
  59. [61]Wicker N Perrin GR Thierry JC Poch O 2001 Secator: a program for inferring protein subfamilies from phylogenetic trees. Mol Biol Evol 18 1435 1441 11470834
  60. [62]Gribskov M McLachlan AD Eisenberg D 1987 Profile analysis: detection of distantly related proteins. Proc Natl Acad Sci USA 84 4355 4358 3474607 PMC305087
  61. [63]Thompson JD Gibson TJ Plewniak F Jeanmougin F Higgins DG 1997 The CLUSTAL_X windows interface: flexible strategies for multiple sequence alignment aided by quality analysis tools. Nucl Acids Res 25 4876 4882 9396791 PMC147148
  62. [64]Vingron M Sibbald PR 1993 Weighting in sequence space: a comparison of methods in terms of generalized sequences. Proc Natl Acad Sci USA 90 8777 8781 8415606 PMC47443
  63. [65]Dosztányi Z Csizmok V Tompa P Simon I 2005 IUPred: web server for the prediction of intrinsically unstructured regions of proteins based on estimated energy content. Bioinformatics 21 3433 3434 15955779
  64. [66]Thompson JD Holbrook SR Katoh K Koehl P Moras D 2005 MAO: a Multiple Alignment Ontology for nucleic acid and protein sequences. Nucleic Acids Res 33 4164 4171 16043635 PMC1180671
  65. [67]Larkin MA Blackshields G Brown NP Chenna R McGettigan PA 2007 Clustal W and Clustal X version 2.0. Bioinformatics 23 2947 2948 17846036
  66. [68]Subramanian AR Kaufmann M Morgenstern B 2008 DIALIGN-TX: greedy and progressive approaches for segment-based multiple sequence alignment. Algorithms Mol Biol 3 6 18505568 PMC2430965
  67. [69]Lassmann T Frings OS Sonnhammer EL 2009 Kalign2: high-performance multiple alignment of protein and nucleotide sequences allowing external features. Nucleic Acids Res 37 858 865 19103665 PMC2647288
  68. [70]Katoh K Toh H 2008 Recent developments in the MAFFT multiple sequence alignment program. Brief Bioinform 9 286 298 18372315
  69. [71]Edgar RC 2004 MUSCLE: a multiple sequence alignment method with reduced time and space complexity. BMC Bioinformatics 5 113 15318951 PMC517706
  70. [72]Notredame C Higgins DG Heringa J 2000 T-Coffee: A novel method for fast and accurate multiple sequence alignment. J Mol Biol 302 205 217 10964570
  71. [73]Do CB Mahabhashyam MS Brudno M Batzoglou S 2005 ProbCons: Probabilistic consistency-based multiple sequence alignment. Genome Res 15 330 340 15687296 PMC546535

Cited by

Sorted by citations

Authors
Thompson, J. D., Linard, B., Lecompte, O., & Poch, O.
Affiliations
Centre National de la Recherche Scientifique (FR)
Corresponding author
Thompson, Julie D.
ORCID
Venue
PLoS ONE
Publisher
Public Library of Science (PLoS)
Published
2011-03-31
Volume
6
Issue
3
Pages
e18093
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0018093
PMC
PMC3069049
OpenAlex
W2092672051
Cited by
242
References
1
Citations by year
2026: 10 · 2025: 14 · 2024: 13 · 2023: 13 · 2022: 13 · 2021: 18 · 2020: 21 · 2019: 12 · 2018: 21 · 2017: 22 · 2016: 12 · 2015: 18 · 2014: 15 · 2013: 13 · 2012: 16
FWCI
6.70
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion, other-oa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Agence Nationale de la Recherche
  • Institut National de la Santé et de la Recherche Médicale
  • Centre National de la Recherche Scientifique
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Keywords
multiple sequence alignment, protein sequence alignment, benchmark test set, divergent sequences, protein families, intrinsically disordered regions, protein motifs, evolutionary systems biology
MeSH terms
Amino Acid Sequence, Proteins, Sequence Alignment, Sequence Homology, Amino Acid, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review