CoChat
Sign Up
PreprintCited by 67Protein Structure and DynamicsMachine Learning in BioinformaticsGenomics and Phylogenetic Studies

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Protein language models trained on multiple sequence alignments learn phylogenetic relationships

Open access
Lupo, Umberto, Sgarbossa, Damiano, Bitbol, Anne-Florence·2022·Nature Communications
DOI 10.1101/2022.03.29.486219PMC PMC9588007

Preprint — published as 10.1038/s41467-022-34032-y · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Self-supervised neural language models with attention have recently been applied to biological sequence data, advancing structure, function and mutational effect prediction. Some protein language models, including MSA Transformer and AlphaFold’s EvoFormer, take multiple sequence alignments (MSAs) of evolutionarily related proteins as inputs. Simple combinations of MSA Transformer’s row attentions have led to state-of-the-art unsupervised structural contact prediction. We demonstrate that similarly simple, and universal, combinations of MSA Transformer’s column attentions strongly correlate with Hamming distances between sequences in MSAs. There-fore, MSA-based language models encode detailed phylogenetic relationships. We further show that these models can separate coevolutionary signals encoding functional and structural constraints from phylogenetic correlations reflecting historical contingency. To assess this, we generate synthetic MSAs, either without or with phylogeny, from Potts models trained on natural MSAs. We find that unsupervised contact prediction is substantially more resilient to phylogenetic noise when using MSA Transformer versus inferred Potts models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2022.03.29.486219

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. 10.1038/nrg3414 · DOI · Paper page
  2. 10.1088/1361-6633/aa9965 · DOI
  3. Bahdanau, D. , Cho, K. & Bengio, Y. Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015). arXiv (2014). URL https://doi.org/10.48550/arXiv.1409.0473. · DOI · Paper page
  4. 2017, Attention is all you need, Advances in Neural Information Processing Systems, 30, 5998
  5. Elnaggar, A. et al. ProtTrans: Towards cracking the language of life’s code through self-supervised learning. bioRxiv (2020). URL https://doi.org/10.1101/2020.07.12.199554. · DOI
  6. Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. U.S.A. 118 (2021). URL https://www.pnas.org/content/118/15/e2016239118. · DOI · Paper page
  7. Rao, R. , Meier, J. , Sercu, T. , Ovchinnikov, S. & Rives, A. Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations (2021). URL https://openreview.net/forum?id=fylclEqgvgd. · DOI · Paper page
  8. Choromanski, K. et al. Rethinking attention with Performers. In International Conference on Learning Representations (2021). URL https://openreview.net/forum?id=Ua6zuk0WRH.
  9. Madani, A. et al. ProGen: Language modeling for protein generation. bioRxiv (2020). URL https://doi.org/10.1101/2020.03.07.982272. · DOI
  10. Madani, A. et al. Deep neural language modeling enables functional protein generation across families. bioRxiv (2021). URL https://doi.org/10.1101/2021.07.18.452833. · DOI
  11. 2022, Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention, Pac Symp Biocomput, 27, 34
  12. 10.1126/science.abj8754 · DOI · Paper page
  13. Chowdhury, R. et al. Single-sequence protein structure prediction using language models from deep learning. bioRxiv (2021). URL https://doi.org/10.1101/2021.08.02.454840. · DOI · Paper page
  14. 10.1038/nsb0295-171 · DOI
  15. 10.1038/nature03991 · DOI
  16. 10.1093/bioinformatics/btm604 · DOI
  17. 10.1016/j.cell.2009.07.038 · DOI
  18. 1999, Statistics in molecular biology and genetics – IMS Lecture Notes – Monograph Series, 33, 236
  19. 10.1073/pnas.0805923106 · DOI · Paper page
  20. 10.1371/journal.pone.0028766 · DOI · Paper page
  21. 10.1073/pnas.1111471108 · DOI · Paper page
  22. 10.1073/pnas.1207864109 · DOI
  23. 2013, Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models, Phys. Rev. E, 87, 012707, 10.1103/PhysRevE.87.012707 · DOI · Paper page
  24. 2014, Fast pseudolikelihood maximization for direct-coupling analysis of protein structure from many homologous amino-acid sequences, J. Comput. Phys, 276, 341, 10.1016/j.jcp.2014.07.024 · DOI
  25. 10.1093/molbev/msy007 · DOI
  26. 2018, Assessment of hard target modeling in CASP12 reveals an emerging role of alignment-based contact prediction methods, Proteins, 86 Suppl 1, 97
  27. 2021, MSA Transformer, Proceedings of the 38th International Conference on Machine Learning, 139, 8844
  28. Meier, J. et al. Language models enable zero-shot prediction of the effects of mutations on protein function. In Beygelzimer, A. , Dauphin, Y. , Liang, P. & Vaughan, J. W. (eds.) Advances in Neural Information Processing Systems (2021). URL https://openreview.net/forum?id=uXc42E9ZPFs. · DOI · Paper page
  29. 2022, Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins, Cell Systems, 13, 274, 10.1016/j.cels.2022.01.003 · DOI
  30. Hawkins-Hooker, A. , Jones, D. T. & Paige, B. MSA-conditioned generative protein language models for fitness landscape modelling and design. In Machine Learning for Structural Biology Workshop, NeurIPS (2021). URL https://www.mlsb.io/papers_2021/MLSB2021_MSA-Conditioned_Generative_Protein_Language.pdf.
  31. Gordon, Y. On Milman’s inequality and random subspaces which escape through a mesh in Rn . In Lindenstrauss, J. & Milman, V. D. (eds.) Geometric Aspects of Functional Analysis, 84–106 (Springer, Berlin, Heidelberg, 1988). URL https://doi.org/10.1007/BFb0081737. · DOI
  32. 10.1371/journal.pone.0013124 · DOI
  33. 10.1126/science.aba3304 · DOI · Paper page
  34. 2020, A primer in BERTology: What we know about how BERT works, Transactions of the Association for Computational Linguistics, 8, 842, 10.1162/tacl_a_00349 · DOI · Paper page
  35. Vig, J. et al. BERTology meets biology: Interpreting attention in protein language models. In International Conference on Learning Representations (2021). URL https://openreview.net/forum?id=YWtLZvLmud7. · DOI · Paper page
  36. Schwaller, P. , Hoover, B. , Reymond, J. L. , Strobelt, H. & Laino, T. Extraction of organic chemistry grammar from unsupervised learning of chemical reactions. Sci Adv 7 (2021). URL https://doi.org/10.1126/sciadv.abe4166. · DOI
  37. 2018, Deep generative models of genetic variation capture the effects of mutations, Nat. Methods, 15, 816, 10.1038/s41592-018-0138-4 · DOI · Paper page
  38. 10.1038/s41467-019-13633-0 · DOI · Paper page
  39. 2021, The generative capacity of probabilistic protein sequence models, Nat. Commun, 12, 6302, 10.1038/s41467-021-26529-9 · DOI
  40. 10.1073/pnas.1711913115 · DOI
  41. 10.1371/journal.pcbi.1006223 · DOI
  42. Rodriguez Horta, E. , Barrat-Charlaix, P. & Weigt, M. Toward inferring Potts models for phy-logenetically correlated sequence data. Entropy 21 (2019). URL https://www.mdpi.com/1099-4300/21/11/1090. · DOI
  43. Rodriguez Horta, E. & Weigt, M. On the effect of phylogenetic correlations in coevolution-based contact prediction in proteins. PLoS Comput. Biol. 17 (2021). URL https://doi.org/10.1371/journal.pcbi.1008957. · DOI
  44. 10.1006/jmbi.1996.0167 · DOI
  45. Hockenberry, A. J. & Wilke, C. O. Phylogenetic weighting does little to improve the accuracy of evolutionary coupling analyses. Entropy 21 (2019). URL https://doi.org/10.3390%2Fe21101000. · DOI
  46. 2020, Coevolutionary Analysis of Protein Subfamilies by Sequence Reweighting, Entropy, 21, 1127
  47. 2022, Extracting phylogenetic dimensions of coevolution reveals hidden functional signals, Scientific Reports, 12, 820, 10.1038/s41598-021-04260-1 · DOI
  48. Weinstein, E. N. , Amin, A. N. , Frazer, J. & Marks, D. S. Non-identifiability and the blessings of misspecification in models of molecular fitness and phylogeny. bioRxiv (2022). URL https://doi.org/10.1101/2022.01.29.478324. · DOI
  49. Sgarbossa, D. , Lupo, U. & Bitbol, A.-F. Generative power of a protein language model trained on multiple sequence alignments. bioRxiv (2022). URL https://doi.org/10.1101/2022.04.14.488405. · DOI · Paper page
  50. 2020, Pfam: The protein families database in 2021, Nucleic Acids Research, 49, D412
  51. Devlin, J. , Chang, M.-W. , Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186 (Association for Computational Linguistics, Minneapolis, Minnesota, 2019). URL https://aclanthology.org/N19-1423.
  52. Gehring, J. , Auli, M. , Grangier, D. , Yarats, D. & Dauphin, Y. N. Convolutional sequence to sequence learning. In Precup, D. & Teh, Y. W. (eds.) Proceedings of the 34th International Conference on Machine Learning, vol. 70 of Proceedings of Machine Learning Research, 1243–1252 (PMLR, 2017). URL https://proceedings.mlr.press/v70/gehring17a.html.
  53. Ho, J. , Kalchbrenner, N. , Weissenborn, D. & Salimans, T. Axial attention in multidimensional transformers. arXiv (2019). URL https://doi.org/10.48550/arXiv.1912.12180. · DOI · Paper page
  54. Ba, J. L. , Kiros, J. R. & Hinton, G. E. Layer normalization. arXiv (2016). URL https://doi.org/10.48550/arXiv.1607.06450. · DOI
  55. 10.1002/(SICI)1099-1255(199611)11:6<619::AID-JAE418>3.0.CO;2-1 · DOI
  56. Seabold, S. & Perktold, J. statsmodels: Econometric and statistical modeling with Python. In 9th Python in Science Conference (2010). URL https://doi.org/10.25080/Majora-92bf1922-011. · DOI

Metadata

Authors
Lupo, U., Sgarbossa, D., & Bitbol, A.-F.
Affiliations
SIB Swiss Institute of Bioinformatics (CH)
Corresponding author
  • Lupo, Umberto
  • Bitbol, Anne-Florence
ORCID
Venue
Nature Communications
Publisher
openRxiv
Published
2022-03-30
Volume
13
Issue
1
Pages
6298-6298
ISSN
2041-1723 (ISSN-L)
DOI
10.1101/2022.03.29.486219
PMC
PMC9588007
OpenAlex
W4307068738
Cited by
67
References
1
Citations by year
2026: 19 · 2025: 21 · 2024: 16 · 2023: 9 · 2022: 2
FWCI
3.92
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Funders
European Commission (851173, HORIZON2020)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
protein language models, multiple sequence alignment, phylogenetic relationships, Hamming distance, coevolution, Potts model, protein contact prediction, historical contingency
MeSH terms
Algorithms, Language, Phylogeny, Proteins, Sequence Alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review